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Introduction 


Christian Zinke-Wehlmann® and Julia Friedrich® 
Institut für Angewandte Informatik (InfAI) e. V., Kompetenzzentrum Künstlich 
Menschlich Intelligenz, Leipzig, Deutschland 
{zinke, friedrich}@infai.org 


Zusammenfassung. Technologische Innovationen verändern die Art und 
Weise, wie Gesellschaften funktionieren. Sie bieten bspw. einen besseren Zu- 
gang zu Wissen, eine individuellere Produktentwicklung, ermöglichen es, neue 
Geschäftsfelder zu erschließen oder Bürokratie abzubauen. Folglich geht die 
digitale Transformation in Organisationen über die bloße Übernahme inno- 
vativer Technologien wie künstliche Intelligenz hinaus. Sie erstreckt sich auf 
die gesamte Organisation, einschließlich ihrer Strukturen, Prozesse und Mit- 
arbeitender. Besonders kleine und mittlere Unternehmen (KMU) stehen von 
der Herausforderung diese Technologien zu adaptieren und Möglichkeiten zu 
finden, diese mit zu gestalten - oft fehlen hier aber die Ressourcen. Jüngste 
Entwicklungen wie die EU-Verordnungen über künstliche Intelligenz (KD und 
Krisen wie unterbrochene Lieferketten, Fachkräftemangel und Klimawandel 
haben die Situation für Unternehmen noch schwieriger gemacht. Das Konzept 
der Industrie 5.0, das als nächster evolutionärer Schritt in der Entwicklung der 
Industrie angesehen wird, zielt darauf ab, diese Anforderungen zu erfüllen. 
Industrie 5.0 wird als zukunftssicherer, widerstandsfähiger, nachhaltiger und 
menschenzentrierter beschrieben. Menschenzentrierte Ansätze sollten als 
Haupttreiber und Innovationsfaktor für die Industrie 5.0 anerkannt werden. 
Ein menschenzentrierter Ansatz für die digitale Transformation erfordert, die 
Bedürfnisse und Interessen der Menschen stärker zu berücksichtigen anstatt 
alleinig auf die Technologie zu fokussieren. Das Ziel der AI Tomorrow Kon- 
ferenz ist es, diesen Diskurs zu fördern, indem die nachhaltige und menschen- 
zentrierte Perspektive der digitalen Transformation in den Mittelpunkt der 
wissenschaftlichen Diskussion gestellt werden. Die erste AI Tomorrow-Konfe- 
renz bot Wissenschaftler:innen unterschiedlicher Forschungsdisziplinen Raum, 
Herausforderungen und Ideen für die Zukunft der KI zu diskutieren. 


Robots and algorithms, data and metaverse — these seem to be what the future is made 
of. While programmers and data scientists build the fundament of a data dominated 
world, the discussion of ethical and ecological impacts for society and economy often 
only come in second. We, the editors of this proceeding, think that both, innovation 
and impact should both be well considered in parallel. Therefore, the scientific com- 
munity should always keep a holistic view on AI which includes trying to anticipate 
possible outcomes and consider the impact of AI for the way we live, we work and 
communicate. With our very first AI Tomorrow conference, we wanted to create a 
space, where scientists from different fields can come together and discuss their ideas 
for the future of Al. 
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The first working conference on Artificial Intelligence Development for a Resi- 
lient and Sustainable Tomorrow (www.ai-tomorrow.org/) took place from 29th to 30th 
of June 2023. Its aim was to bring together scientists from various fields that focus on 
more than the technological aspects of AI development. We wanted to shed light on 
the opportunities and challenges of AI with regard to resilience and sustainability of 
business models, societies and the environment. 

Digital innovations change the way in which societies work. On the one hand, the 
increasing amount of digital devices and their performance capacity bring new oppor- 
tunities to all aspects of our live, e.g. in terms of travel planning due to smart route 
planning and ticket booking, the reduction of red tape, e.g. through digital identity 
cards, and or even inclusion processes, e.g. through text-to-speech applications or 
semi-automated translation or summarizing of texts. Likewise, the increasing number 
of Al-based tools and methodologies change the world of businesses. This includes 
improved knowledge access, more extensive product development and the creation of 
new business areas. The speed in which processes and technologies change is challen- 
ging for all. Companies all over the world must adapt to the rapid development, which 
requires a multidisciplinary approach to digital transformation to remain competitive. 

One particular focus of our conference is on small and medium-sizes companies 
(SME). In Germany, SME have a major impact on the economy. In fact, the share of 
SMEs in Germany was 99.4 percent in 2019.! They account for the majority of all 
apprenticeships, making them one of the main drivers of competence development.” 
While large companies have the financial abilities to hire change manager and invest 
in consulting and training, SMEs struggle to keep up with the speed of the techno- 
logical development because they simply don’t have the financial and personal re- 
sources for adaption. To compete with larger businesses in terms of affordable prices 
and high-quality products, SMEs need to adapt to new digital innovations. The use of 
artificial intelligence (AI) — one of the key technologies of digitalization — can have 
a significant impact on business success and is increasingly becoming a game chan- 
ger for SMEs. In contrast, Accenture’s 2022 study? shows that only twelve percent of 
companies consider their AI maturity level as high enough to use it to grow and trans- 
form their business, while 63 percent are still in the experimental phase of adopting 
Al in the enterprise. 


Statistisches Bundesamt (2019): Shares of small and medium-sized enterprises in selected 
variables. URL: https://www.destatis.de/EN/Themes/Economic-Sectors-Enterprises/Enterpri- 
ses/Small-Sized-Enterprises-Medium-Sized-Enterprises/Tables/total-cik.html, last accessed 
1° September 2023. 
2 Federal Ministry for Economic Affairs and Energy (2019): SMEs digital. Strategies for the 
digital transformation. URL:  https://www.bmwk.de/Redaktion/EN/Publikationen/Mittel- 
stand/smes-digital-strategies-for-digital-transformation.pdf?__blob=publicationFile&v=5, 
last accessed 1° September 2023. 
Accenture (2022): The art of AI maturity: Advancing from practice to performance. URL: 
https://www.accenture.com/_acnmedia/Thought-Leadership-Assets/PDF-5/Accenture-Art-of- 
AlMaturity-Report.pdf, last accessed 1°! September 2023. 
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In addition to the challenges of digital transformation, current developments and 
crises such as EU AI regulations, disrupted supply chains, skilled workers shortage 
and climate change, have broadened the focus and question the effects and opportu- 
nities of digitization with a view to the next evolutionary step in the development of 
Industry 4.0, namely Industry 5.0. In a 2021 policy brief, the European Commission 
describes Industry 5.0 as “more future-proof, resilient, sustainable, and people-cen- 
tric”.* While Industry 4.0 increases resilience and “the positive impact on the ecolo- 
gical and economic dimensions seems to be fairly widespread”, the social dimension, 
especially human-centric approaches and measures, may still be underestimated. Peo- 
ple or human-centric approaches are therefore the main driver and innovation factor 
for Industry 5.0. A human-centered approach to digital transformation means focusing 
on people’s needs and interests rather than viewing technology as the sole actor. 

With this focus on human-centred approaches in AI development; a multi- 
dimensional value discussion about impacts and sustainability of AI; measurements 
for AI values; concrete use cases of AI development (in SME) the AI Tomorrow Con- 
ference brings experts from various fields together. It is the goal of the conference and 
our hope to foster an interdisciplinary dialogue and identify opportunities for collabo- 
ration and advancement in the responsible development and deployment of AI. 

The findings of the articles in this proceeding have been presented at the 1° Con- 
ference on Artificial Intelligence Development for a Resilient and Sustainable Tomor- 
row (AI Tomorrow) which was hold an 29" and 30" of June 2023 in Leipzig. A spe- 
cial thanks go to the session chairs, Dr. Sandra Schumann, Anja Brückner, Dr. Amit 
Kirschenbaum and Jasmin Mayan. Their support in form of moderation and organiza- 
tion of their sessions was a crucial factor in making the 1° AI Tomorrow Conference a 
success. 

Except for the positioning paper which sums up the first conference keynote, 
given by Dr. Christian Zinke-Wehlmann, the articles in the proceedings have been 
peer reviewed by the members of the scientific board in a double-blind review pro- 
cess. 

Leipzig, Germany 

28.08.2023 


4 European Commission, Directorate General for Research and Innovation (2021): Industry 
5.0: towards a sustainable, human centric and resilient European industry. Publications Office 
of the European Union. 
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Resilient and Sustainable AI. Positioning 
paper on the relation of AI, resilience 
and sustainability 


Dr.Christian Zinke Wehlmann\™ © 


Institut für Angewandte Informatik (InfAI) e. V., Kompetenzzentrum Kiinstlich 
Menschlich Intelligent, Goerdelerring 9, 04109 Leipzig, Germany 
zinke@infai.org 


Abstract. In the contemporary debate, surrounding the future of work and life, 
Artificial Intelligence (AD), resilience, and sustainability have emerged as pivo- 
tal concepts. Within the industrial realm, their collective convergence is driving 
unprecedented transformative shifts, challenging traditional paradigms. This 
positioning paper delves into the intricate interlinkages binding these three pa- 
radigms. Examples such as Al-driven automation, enhancing efficiency, and 
predictive maintenance, reducing machinery downtime, underscore the trans- 
formative role of AI in the industry. Meanwhile, an increasing emphasis on en- 
vironmental responsibility highlights the growing importance of sustainability 
in the industrial sector. Resilience, embodied through the ability to withstand 
crises and maintain strong supply chains, is equally essential. The article also 
delves deep into the specific relations between AI, sustainability and resilience. 
By weaving these concepts together, the paper aims to provide a holistic per- 
spective on the interconnectedness, emphasizing the need for a balanced ap- 
proach in the modern industry to ensure not only technological advancement 
but also a resilient and sustainable future. 


Keywords: Artificial Intelligence - Resilience - Sustainability 
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Artificial intelligence, resilience, and sustainability are major concepts of the current 
debate on the future of work and life. In the industrial landscape, the convergence of 
AI, sustainability, and resilience is ushering in transformative changes that will res- 
hape traditional paradigms. Each of these elements itself, and to an even greater extent 
together, are emerging as key drivers for the next era of industrial growth. Some well- 
known examples of the pivotal role of AI in Industry are: 


© Der/die Autor(en) 2024 

C. Zinke-Wehlmann und J. Friedrich (Hrsg.): First Working Conference on Artificial 
Intelligence Development for a Resilient and Sustainable Tomorrow, AIDRST 2023, 
Informatik aktuell, S. 5-19, 2024. https://doi.org/10.1007/978-3-658-43705-3_2 


Automation: Al-driven automation streamlines processes, reduces waste, and opti- 
mizes resource use, and thus leading to enhanced efficiency and productivity. E.g. 
a study of Brynojolfsson et al. [8] found that generative AI increases the work of 
customer support agents significantly. 
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e Predictive maintenance: Through analysis, AI identifies and predicts potential ma- 
chinery breakdowns before they occur, reducing downtime and ensuring smooth 
operations [11]. 


An example for the crucial impact of sustainability on industry is: 


e Environmental Responsibility: Industrial sectors are under increasing pressure to 
align with global environmental goals. Implementing sustainable practices can 
drastically reduce the carbon footprint and resource wastage. A report by the Busi- 
ness and Sustainable Development Commission [9] highlighted that embracing 
sustainability could unlock $12 trillion in new market value by 2030. 


To underline the concept of Resilience, well know examples of usage are: 


e Withstand Crisis: In an ever-changing global landscape filled with economic uncer- 
tainties, geopolitical tensions, and the looming threat of pandemics, industries must 
prioritize resilience to manage risks. According to Deloitte [12], resilient organiza- 
tions! are 3 times more likely to outperform their industry peers during a crisis. 

e Supply Network Strength: Resilience ensures continuity in supply chains, even 
amidst disruptions. A resilient system can quickly adapt and recover from shocks, 
exemplarily emphasized by [31]. 


For the modern industry, AI, sustainability, and resilience aren’t just buzzwords — they 
are essential pillars that will define their growth, reputation, and longevity in the co- 
ming decades. As these concepts may be deeply interwoven, industries must prioritize 
their simultaneous development and integration. This article introduces these three 
concepts and analyses the way in which they are linked together. Examples for this 
interlinkage are AI algorithms that supports sustainable energy industry [1] or AI-dri- 
ven data analytics that can be pivotal in developing sustainable supply chains by pre- 
dicting potential disruptions and ensuring a stable, resilient system [22]. It is no aim 
to provide a comprehensive overview of the state of research, but to sharpen the terms 
and highlight their mutual interdependencies. 

Therefore, this positioning paper delves into the intricate relationships binding 
AI, resilience, and sustainability. We aim to elucidate the underpinnings of resilient 
and sustainable AI systems — with a holist point of view [26] — and finally provide a 
cohesive point of view that interlinks all three paradigms. By charting this tripartite 
connection, we hope to lay a foundation for future research and guide AI’s trajectory 
towards a future that is not just technologically advanced, but also resilient and sustai- 
nable. 


! The 2021 Deloitte Global Resilience Report summarizes attributes like preparedness, adapta- 
bility, collaboration, trustworthiness, and responsibility for resilient organizations. 
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Before we can describe the interlinkage between the main concepts, we need to de- 
fine the terms. After introducing the term AI, we will have a closer look on resilience 
as well sustainability. 


2 What is AI? 


Even though everybody talks or writes about AI, most people have no concrete idea 
of what the term actually stands for. To talk about artificial intelligence, it is therefore 
necessary to define it in the first place. In the current period, with new software pro- 
ducts based on machine learning (ML) technology appearing almost daily, ML domi- 
nates the public’s perception and broad understanding of what Al is. 

In a JRX technical report by the EU [16], various definitions were taken up and 
analysed by AI Watch with the goal of sharpening the vague concept of AI. It conclu- 
des in stating, that AI is a catch-all term and can mean different things: 


“AI is a generic term that refers to any machine or algorithm that is capable of 


e observing its environment, 

e learning, 

e and based on the knowledge and experience gained, taking intelligent action or 
proposing decisions.” [16] 


This definition approach essentially builds on the idea of a description of the capabi- 
lities of the technology. This means that the AI term can thus encompass any techno- 
logy that is capable of observing, learning, and, based on built-up knowledge, making 
decisions, and recommending or implementing them. Hence, we can see that multiple 
technologies fall under such a broad AI definition. 

AI Watch develops an AI taxonomy based on this definition to cluster various AI 
approaches. Core areas of AI (Reasoning, Planning, Learning, Communication, Per- 
ception) are distinguished from transversal areas (Integration & Interaction, Services, 
Ethics & Philosophy) and sub-areas are named, such as knowledge representation in 
the area of reasoning [16]. Beside the more or less academic discourse about the ques- 
tion “What is AI?”, AI is getting to practice, which results in a political as well as 
legal definition of AI. 

In the course of developing rules for AI (“LAYING DOWN HARMONISED 
RULES ON ARTIFICIAL INTELLIGENCE (ARTIFICIAL INTELLIGENCE ACT) 
AND AMENDING CERTAIN UNION LEGISLATIVE ACTS”) by the EU, three 
technologies are fundamentally distinguished: 


“(a) Machine learning approaches, including supervised, unsupervised and reinforcement 
learning, using a wide variety of methods including deep learning; 
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(b) Logic- and knowledge-based approaches, including knowledge representation, induc- 
tive (logic) programming, knowledge bases, inference and deductive engines, (symbolic) 
reasoning and expert systems; 

(c) Statistical approaches, Bayesian estimation, search and optimization methods.” [15] 


Both presented definitions are highly overlapping and demonstrate somehow the close 
relation between legal/political interpretation and a specific academic debate. Again, 
this paper does not aim to present a full history of Al or a literature-based analysis of 
the current AI definition debate. From the authors point of view, the presented defini- 
tions are a practical ground for current developments of AI (Europe) in research and 
industry because of its role as legal framework. It sets the basics to understand that the 
term “AT” is generic, but refers to specific properties of technical systems, which can 
manifest completely differently — as a robot or as a statistical model. The expressions 
and intensity of the abilities are not further defined here. It describes what a techni- 
cal system does (observe, learn, propose or implement decisions based on collected 
knowledge) to determine if this system is artificially intelligent. 


3 What is Resilience? 


Resilience is a term that is used in many contexts, e. g. psychology. It probably ori- 
ginally stems from the field of physics [25]. In the following, we will refer to eco- 
nomic resilience, which, likewise, can have many different forms. Resilience occurs 
within companies, as well as within value-added networks, and entire economies. The 
term was first used by Holling; stating that economic resilience “refers to the extent to 
which a shock can be absorbed by a local stable domain before it is induced into some 
other stable equilibrium” (Modica & Reggiani, 2015). Looking at the term through a 
systemic lens (e.g., in addition to 6), resilience is the ability of a system, 


e to ensure the system’s stability despite (massive) external (or internal) disturban- 
ces/crises, 
e to adapt to permanent changes caused by crises. 


Similar to the AI definitions, the given approach to define resilience bases on an ab- 
ility of a system (in this case, not necessarily a technical system). Unlike attributing 
the what a system must do (see AJ), this definition rather focuses on the consequen- 
ces or the result. This definition stands in contrast to the definition of organizational 
resilience in standardization. E.g., in the ISO standards’ definition, resilience isn’t an 
element of corporate control but emerges from the application of these control ele- 
ments. [19] The question is whether a result of applying unspecified mechanisms can 
be a capability of systems? The aforementioned definition suggests precisely this — 
but above all, it suggests that systems can exhibit mechanisms that produce resilience. 
To address this challenge, resilience can be defined as responsiveness: 
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“Resilience is the responsiveness of the system, i.e., its elasticity or capacity to rebound 
after a shock, indicated by the degree of flexibility, persistence of key functions, or ability 
to transform [...]” [27] 


The mechanisms to achieve this can be termed as a resilience strategy. This, essen- 
tially, describes the deliberate approach to achieving resilience and can be clustered 
into three areas: 


e Preparation and prevention of shocks/disturbances or crises, 
e Mitigation and prevention of consequences from the disturbance/crisis, 
e Adjustments to the crisis (consequences of the crisis). 


If a system possesses mechanisms that serve this purpose and are functional, then a 
system can be described as resilient. Such mechanisms are, for instance, part of sup- 
ply chain research, which, for example, views “Situation Awareness” as a metric for 
“Readiness” as preparation see [19]. On the other hand, there are a number of stan- 
dards that deal with resilience strategies. For companies, there’s ISO 22316, which 
suggests what to do or exactly how to achieve resilience, e.g., that the management 
should establish relevant business resilience goals and formulate assessment criteria 
for evaluating resilience attributes [21] 

In addition to these standards, there are several others, closely tied to this capabi- 
lity, such as Compliance Management System (CMS) according to ISO 37301, Risk 
Management: ISO 31000, Knowledge Management: DIN EN ISO/IEC 27.001:2017; 
ISO 30301:2019; DIN ISO 30401:2021, Supply Chains: ISO 28000; DIN ISO 
20400:2021 ISO/TS 22318. As already explained in the introductory words, this pre- 
sentation is not comprehensive but provides a brief insight without even approaching 
the state of research. 


4 What is Sustainability? 


After AI and resilience, sustainability remains to be addressed, an equally complex 
term. From the authors perspective, the concept of sustainability only becomes mea- 
ningful when it is understood as a sustainable development, meaning a process — and 
this is also how some other authors perceive it. Sustainability itself is a “paradigm for 
thinking about the future in which environmental, societal, and economic considerati- 
ons are balanced in the pursuit of improved quality of life. The ideals and principles 
behind it rest on broad concepts such as intergenerational equity, gender equity, social 
tolerance, poverty alleviation, environmental preservation and restoration, natural re- 
source conservation, and building just and peaceful societies” [23]. 

In the context of the so-called Triple Bottom Line and from a business perspec- 
tive, sustainability is often understood as follows: 


e Sustainability seeks to evaluate the corporation’s financial, social, and environmental outco- 
mes over a duration. Only an enterprise that adopts a TBL approach truly acknowledges the 
comprehensive expenses associated with its operations. [13] 
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However, the “inventor” of the TBL, John Elkington, clarified in 2018 that, “[...] the 
original idea was broader still, urging businesses to track and manage economic (not 
just financial), social, and environmental value added — or destroyed.” [14] Thus, sus- 
tainability can’t just be measured by profit and loss, but by the well-being of people, 
now and in the future. The TBL emphasized radical transformation, market disrup- 
tion, asymmetric growth (wherein non-sustainable sectors are deliberately marginali- 
zed), and the amplification of emergent market strategies (Elkington, 2018). Moreo- 
ver, the TBL can be understood as a hierarchical system, not an equal one, because: 
Ecological conditions and resources serve as foundational elements for both life and 
societal structures. A well-operating society is pivotal for addressing social necessities 
and instituting an economic framework. This economic structure facilitates both re- 
source utilization and prosperity, catering to material requirements [20]. 

From the political debate on sustainability, the concept of sustainable development 
marks a milestone. Sustainable development is a process filled with life in the con- 
text of the UN, especially the Brundtland Commission, which states that sustainable 
development addresses contemporary requirements while ensuring that future genera- 
tions retain the capacity to fulfill their respective needs [5]. Without delivering a com- 
plete historical account (there’s plenty of literature for that, e.g., [18]), these develop- 
ments culminated in the Sustainable Development Goals, the UN’s 17 sustainability 
targets. These are: 


#1 No Poverty 

#2 No Hunger 

#3 Good Health and Well-being 

#4 Quality Education 

#5 Gender Equality 

#6 Clean Water and Sanitation 

#7 Affordable and Clean Energy 

#8 Decent Work and Economic Growth 
#9 Industry, Innovation, and Infrastructure 
#10 Reduced Inequalities 

#11 Sustainable Cities and Communities 
#12 Responsible Consumption and Production 
#13 Climate Action 

#14 Life Below Water 

#15 Life on Land 

#16 Peace, Justice, and Strong Institutions 
#17 Partnerships for the Goals 


Combining these objectives with the TBL, goals 6, 13, 14, 15 (environment) can be 
seen as the foundation for goals 1, 2, 3, 4, 5, 7, 11 and 16 (society), which in turn ena- 
ble goals 8, 9, 10, 12 (economy) and can be achieved through 17 (cooperation). As an 
example gender equality (#5) can be seen as a driver for high quality education (#4), 
enabling decent work and economic (#8) in the long term [24]. However, a too sim- 
plistic understanding would neglect the high interdependence of individual factors, as 
cause and effect relations are often precisely the opposite, e.g., sustainable production 
allows life on land and so on. 
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Having introduced the third term, the next sections will relate two terms to each 
other, finally focusing on the relationship between AI, sustainability, and resilience 
and discussing the role of AI. 


5 Sustainability and Resilience 


Resilience and sustainability seem somewhat contradictory in the presented version. 
While sustainable development takes a very long-term view, resilience is more focu- 
sed on narrow time horizons. This is especially the case when we correctly consider 
the type of response. Resilience is a reactive ability to a very abrupt disruption or 
crisis, unlike sustainable development which focuses on changes observed over very 
long processes. However, both concepts consider the maintenance of systems and 
their performance, but the time horizons they consider are very different. 

Nevertheless, both concepts are related. A sustainable system should be able to 
handle shocks and crises. “In political interpretation, resilience and sustainability are 
two sides of the same coin. Only sustainable value chains remain robust and intact 
in the long run. It is therefore essential that companies keep an eye on the impact on 
people, the environment, and the climate in their activities” (4, translated). The logical 
relationship that arises from this is: “Resilience represents a necessary, but not suffi- 
cient, condition for sustainability.” (7, translated). The logical relationship that arises 
from this is: “Resilience represents a necessary, but not sufficient, condition for sustai- 
nability” (Translated from 7). 

In short: Not every resilient system is sustainable, but every sustainable system is 
resilient. This dependency and interdependence can lead to elements of sustainable 
development, being part of resilience strategies as they can be found in the BMWK- 
Whitepaper in context of Platform Industry 4.0 [3]. Sustainability is one of the levels 
of the resilience strategy, which spans the various impact phases of resilience. The 
social dimension of sustainability — health protection, societal benefit, working condi- 
tions — is particularly emphasized. In other words, in Industry 4.0, social sustainability 
constitutes a sufficient condition for resilience 


6 Aland Resilience 


Resilience and AI can be related in various ways. As already described, AI always 
pertains to a technical system’s ability to learn/act, while resilience addresses a sys- 
tems responsiveness. From this, three potential connections arise: 


e Al systems are used to establish this resilience, making AI an integral part of the 
resilience strategy. 

e Developments in AI systems can cause a shock to companies or industries. 

e The resilience of AI systems itself comes into focus. 
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Regarding the point that Al is used for resilience, there are numerous (German or EU) 


research projects and publications (e.g. [2, 10, 31]). Projects and their areas can be 
taken from the following table (Tab. 1): 


Table 1. Resilience and AI — examples of domains and research projects 


Infrastructure aKtlv: Agile network control to increase the resilience of the critical infras- 
tructure water supply 

BESKID: Fire design simulation in rail vehicles using Al-based data 

| IKIGas: Industrial Artificial Intelligence for safety in gas networks 


Society NEBULA: User-centered Al-based detection of fake news and misinforma- 
tion 
AIFER: Artificial intelligence for analyzing and merging Earth observation 
and internet data for decision support in disaster management 
FAKE-ID: Video analysis using artificial intelligence to detect false and 
| manipulated identities 


Supply chains and LEAS: Land-side recommendation for traffic situations with highly auto- 
organizations mated or autonomous ships 
SPAICER: Minimize production disruptions and interruptions in supply 
chains 
ResKriVer — Communication & Information platform for resilient, crisis- 
relevant supply networks 
PAIRS - Privacy-Aware, Intelligent and Resilient Crisis Management 
KISS - Al-supported Rapid Supply Network 
DAKI-FWS - Data- and Al-supported early warning system 
CoyPu - Cognitive Economy Intelligence Platform for the resilience of 
economic ecosystems 


AI can be used in all phases of the resilience strategy. The most intuitive applicati- 
ons involve Al-supported prediction systems for specific scenarios. This is particularly 
useful when risks for a shock can be foresight — hence, enough data with enough in- 
formation is available to make such predictions. 

Besides AI’s beneficial power to strengthen resilience, AI system may also have 
other roles. The second role of AI is, that AI is realized as a trigger for shocks on in- 
dustries or companies. A publicly discussed example of such kind of shock, was the 
discussion about ChatGPT and its impact on stocks in the education market, which 
(temporarily) plunged’. This example is a prototype for public discourses about the 
impact of AI, which moves between from euphoria to dystopia (in this case near eco- 
nomic dystopia). Reality probably lies somewhere in the middle, as AI development 
influences society in general — just as society and economic systems influence techno- 


? See press: https://www.welt.de/wirtschaft/article245 123004/KI-50-Prozent-Absturz-die-er- 
ste-Branche-droht-ChatGPT-zum-Opfer-zu-fallen.html#cs-lazy-picture-placeholder-01c4ee- 
daca.png. To my knowledge, there is no scientific analysis of the direct connection or evi- 
dence between the two events — however, there is a certain plausibility. 
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logical developments, see also [33]. The interdependencies are complex, making it not 
unlikely that AI developments will cause shocks to hit companies and sectors. 

Hence, it seems logical to examine how resilience can be built up for such case. 
Initially, understanding what Al is and that it will induce change should help organi- 
zations prepare for potential Al-driven shocks. Building this competence or accessing 
educational resources (also pertaining to social sustainability), is useful throughout all 
resilience phases. The level of competency being built also aids during the adaptation 
phase, leveraging the shock for potential adaptations and restructuring measures. 

Lastly, the resilience of AI systems themselves can be examined. Why? Since the 
beginning of computing, various significant errors have happened due to numerical in- 
accuracies. A particularly infamous incident was the self-destruction of an Ariane 5 
rocket in 1996. Such events have driven the establishment of a dedicated research area 
that concentrates on creating and refining resilient algorithms, a niche within the realm 
of numerics. This needs to be extended on AI Systems to prevent the given risks [30]. 

As AI systems increasingly permeate our daily lives and work, it becomes essen- 
tial to scrutinize their resilience to prevent potential harm to social systems or the or- 
ganizations we work in. Wittenbrink et al. [30] tackled this challenge, concluding that 
five factors define resilient AI: 


Safety 

Accuracy 
Reliability 
Robustness 
Comprehensibility 


These points refer, among other things, to the data foundation upon which any AI is 
based, its origin, and conformity (Governance Risk Compliance or GDPR), IT secu- 
rity, and the model’s explainability and resilience, or that of the technical system. Re- 
search around this topic is, to the authors knowledge, still in its infancy, and the ques- 
tion of extending and specifying these points is probably an ongoing process. In the 
future, questions about model transparency and the focus on the common good will 
likely arise. I will revisit the question of AI system resilience when introducing the 
connection between Al and sustainability, as resilience and sustainability are logically 
connected, and some readers might have noticed certain overlaps. 


7 Sustainability and AI 


There are several works about sustainability and AI. As an example, the definition of 
sustainable Al is still being formulated. Bjørlo et al. and Halsband focuses on the idea 
that sustainable AI should meet current needs without compromising the future — in 
addition to the presented definition of sustainability. Some researchers have identified 
boundary conditions for sustainable AI, including diversity, trust, and the capacity for 
self-organization and learning. 

One interesting work is “Sustainability Criteria for Artificial Intelligence” (,,Nach- 
haltigkeitskriterien für künstliche Intelligenz“) authored by Friederike Rohde, Jose- 
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phin Wagner, Philipp Reinhard, Ulrich Petschow, Andreas Meyer, Marcus Voß, and 
Anne Mollen [26]. The paper delves into the realm of artificial intelligence (AD), 
particularly machine learning (ML) and sustainability. The increasing deployment 
of AI systems has sparked debates globally about their societal, environmental, and 
economic impacts. Concerns include non-transparent decision-making processes, di- 
scrimination, rising energy consumption, greenhouse gas emissions during AI model 
development, and broader consequences on labor markets, consumption patterns, and 
the market power of large corporations. The authors aim to provide a perspective on 
sustainable AI, connecting it with discussions on Al ethics, Green Al, and the sustai- 
nability of AI and advocate for an embedded perspective on sustainable AI, viewing 
technology as socially shaped and shapable, not just a neutral tool [26]. 

From a sustainability perspective, AI systems present multiple challenges. One of 
these significant challenges is the interdependency between different SDGs and AI. 
Direct and indirect impacts of AI on SDGs need to be considered’. Vinuesa et al. [28] 
note that AI could aid in achieving 79% of the Sustainable Development Goals, like 
alleviating poverty and improving education. However, it could also hinder 35% of 
these goals by, for instance, consuming vast natural resources, propagating biases 
against gender equality, or bolstering autocratic regimes. While AI in sectors like 
smart manufacturing can boost productivity and conserve resources, it might lead to 
job losses in areas like finance*. However, the work of Rohde et al. (2021) includes a 
systematic overview of impacts throughout the AI lifecycle. Further Rohde et al. pre- 
sents conceptual ideas for a comprehensive sustainability assessment of AI. It intro- 
duces thirteen sustainability criteria and five cross-sectional criteria with correspon- 
ding indicators based on existing scientific and societal discourses on the impacts of 
Al [26]: 


Sustainability Criteria for Artificial Intelligence: 


Ecological Criteria: 


e Energy Consumption: Refers to the amount of energy utilized by AI systems during 
their operation, and further emphasizing the need for energy-efficient solutions. 

e CO2 and Greenhouse Gas Emissions: Highlights the environmental impact of AI 
systems in terms of carbon dioxide and other greenhouse gas emissions, advoca- 
ting for low-emission technologies and solutions. 

e Sustainability Potentials: Discusses the potential of AI to contribute positively to 
environmental sustainability, such as optimizing resource use or aiding in conser- 
vation efforts. 

e Indirect Resource Consumption: Addresses the resources consumed indirectly 
due to AI operations, such as the materials used in hardware manufacturing or the 
water footprint of data centers. 


First benchmarking initiatives like the platform for learning systems (translated) (https:// 
www.plattform-lernende-systeme.de/nachhaltigkeit-karte.html) underlines the importance. 
“Goldman Sachs employed six hundred traders in 2000, the corporation was able to reduce 
their number of human traders to two by 2017 because of advances in narrow AI” [17]. 
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Social Criteria: 


Transparency & Responsibility/Accuracy: Stresses the need for clarity in how Al 
systems operate and make decisions, ensuring that users and stakeholders unders- 
tand and can trust the technology. 

Technical Reliability & Human Oversight: Emphasizes the importance of AI sys- 
tems being reliable and accurate, with mechanisms for human intervention and 
oversight when needed. 

Data Protection & Privacy: Emphasizes the protection of user data and privacy, en- 
suring that AI systems handle personal information responsibly and securely. 
Inclusive and Participatory Design: Advocates for the design and development of 
AI systems that are inclusive of diverse user groups and allow for stakeholder par- 
ticipation in decision-making processes. 

Cultural Sensitivity: Stresses the importance of AI systems being respectful and 
considerate of cultural differences, ensuring that they do not perpetuate biases or 
stereotypes. 


Economic Criteria: 


Distribution Effect in Target Markets: Discusses how AI impacts the distribution of 
resources, wealth, or opportunities in its target markets, emphasizing equitable dis- 
tribution. 

Working Conditions & Working Place: Highlights the impact of AI on the work- 
place, including potential changes in job roles, working conditions, and the nature 
of work. 

Market Diversity & Exploitation of Innovation Potential: Emphasizes the import- 
ance of a diverse market landscape where AI fosters innovation without leading to 
monopolies or stifling competition. 


Cross-sectional Criteria: 


Defined Responsibilities: Organizations should have clear responsibilities for ensu- 
ring the sustainability of Al. 

Code of Conduct: Defines the values and norms for the implementation and use of 
AI systems. 

Stakeholder Analysis & Participation: Involves identifying and integrating stake- 
holders in the AI governance process. 

Documentation of AI Systems: Comprehensive documentation of AI systems, in- 
cluding data sources and methodologies. 

Risk Management: Identifying and managing potential risks associated with AI 
systems. 


With the definition and the naming of criteria for the sustainable use of AI, a first spe- 
cification has been made that helps all organizations to orient themselves. In the fol- 
lowing chapter, we will draw a conclusion and bring together the relationships of the 
triad AI, resilience, and sustainability. 
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8 Conclusion: On the relation of AI, resilience 
and sustainability 


In the previous chapters, numoerous definitions and explanation approaches for the 
understanding of the major concepts of AI, sustainability and resilience and their 
interconnections were given. To sum it up, we can state that AI is a generic term for 
technical systems with special capabilities (observe, learn, act), Resilience refers to a 
responsiveness of a system, and sustainability or sustainable development addresses 
contemporary requirements while ensuring that future generations retain the capacity 
to fulfill their respective needs. 

AI, understood as a technical system, can be resilient, and in an extended sense, 
sustainable (with sustainability implying resilience, as demonstrated). Hence, AI sys- 
tems need to ensure to fulfil or not to violate contemporary requirements on sustaina- 
bility (given above). Therefore, resilient AI systems are a necessary but not sufficient 
condition for sustainable AI systems. Technical systems are integral components of 
socio-technical systems, suggesting that these technical systems impact and are inter- 
dependent with social, ecological, and economic systems throughout the AI systems' 
lifecycle. The sustainability index approach introduced for AI [26] is a pivotal step in 
identifying and measuring these interdependencies. To integrate the aforementioned 
works, the author proposes that the sustainability and inherent resilience of AI systems, 
as well as their impact concerning sustainability factors, are sufficient criteria for eva- 
luating AI’s overall sustainability. From this perspective, it’s unsurprising that many re- 
silient factors are already evident within the presented sustainable AI index, such as the 
data foundation in AI systems documentation and Technical Reliability. This suggests 
that some resilience factors might need more in-depth consideration in terms of [30]: 


e Safety (Hardware, Model, Data) 
e Robustness (e.g., adversarial robustness) 


Moreover, we must recognize that certain AI technologies could be leap innovations 
(market shocks) and might be unpredictable. We could also face challenges when 
trying to identify rebound effects. AI technologies will invariably introduce potential 
shocks to systems. To prepare for these shocks, developing competencies has been 
showcased as an effective way to navigate such potential Al-induced disruptions (and 
perhaps even broader ones), thus fostering sustainable socio-technical systems. As a 
result, the index should also give significantly more weight to: Competency develop- 
ment for co-creating AI systems as a pertinent factor. 

Besides those factors, more generally, we may need to considerate and classify the 
factors in different level. Finally, sustainable AI systems needs to be resilient — it is a 
logical consequence. 


References 


1. Ahmad, T., Zhang, D., Huang, C., Zhang, H., Dai, N., Song, Y., & Chen, H. (2021). Artifi- 
cial intelligence in sustainable energy industry: Status Quo, challenges and opportunities. 
Journal of Cleaner Production, 289, 125834. https://doi.org/10.1016/j.jclepro.202 1.125834. 


W 


10. 


11. 


12. 


13. 


14. 


15: 


16. 


1); 


18. 


19. 


20. 


21. 


Resilient and Sustainable Al. Positioning paper on the relation ... 17 


. Alshater, M. M., Kampouris, I., Marashdeh, H., Atayah, O. F., & Banna, H. (2022). Early 


warning system to predict energy prices: the role of artificial intelligence and machine lear- 
ning. Annals of Operations Research, 1-37. https://doi.org/10.1007/s10479-022-04908-9. 


. BMWK. (2022). Plattform Industrie 4.0: Resilienz im Kontext von Industrie 4.0. 
. BMWK. (2023). „Resilienz und Nachhaltigkeit gehen Hand in Hand“ — OECD-Minister- 


treffen zu verantwortungsvollem unternehmerischem Handeln. https://www.bmwk.de/ 
Redaktion/DE/Pressemitteilungen/2023/02/20230215-resilienz-und-nachhaltigkeit-gehen- 
hand-in-hand.html. Zugegriffen: 1. Sept. 2023. 


. Borowy, I. (2013). Defining sustainable development for our common future: A history of 


the World Commission on Environment and Development (Brundtland Commission). Rout- 
ledge. 


. Brende, B., & Sternfels, B. (2022). Resilience for sustainable, inclusive growth. https:// 


www.mckinsey.com/capabilities/risk-and-resilience/our-insights/resilience-for-sustainable- 
inclusive-growth. Zugegriffen: 1. Sept. 2023. 


. Brinkmann, H., Harendt, C., Heinemann, F., & Nover, J. (2017). Ökonomische Resi- 


lienz—Schlüsselbegriff für ein neues wirtschaftspolitisches Leitbild? Wirtschaftsdienst, 
97(9), 644-650 


. Brynjolfsson, E., Li, D., & Raymond, L. (2023). Generative Al at work. National Bureau of 


Economic Research. 


. Business, Sustainable Development Commission, others. (2017). Better business, better 


world. Systemiq. https://sustainabledevelopment.un.org/content/documents/2399Better- 
BusinessBetterWorld.pdf. 

Charles, V., Emrouznejad, A., & Gherman, T. (2023). A critical analysis of the integration 
of blockchain and artificial intelligence for supply chain. Annals of Operations Research, 
1-41. https://doi.org/10.1007/s10479-023-05 169-w. 

Chui, M., Henke, N., & Miremadi, M. (2018). Most of AI’s business uses will be in two 
areas. Harvard Business Review, 20. 

Deloitte. (2021). Building the Resilient Organization. 2021 Deloitte Global Resilience Re- 
port. Deloitte. 

Elkington, J. (2009). Triple bottom linel It consists of three Ps: Profit, people and planet. 
The Economist. 

Elkington, J. (2018). 25 years ago I coined the phrase “triple bottom line.” Here’s why it’s 
time to rethink it. Harvard Business Review, 25, 2—5. 

European Commission. (2021). Laying down harmonised rules on artificial intelligence (ar- 
tificial intelligence act) and amending certain union legislative acts. European Commission, 
106, 1-108. 

European Commission, Joint Research Centre, Samoili, S., Löpez Cobo, M., Delipetrev, B., 
Martinez-Plumed, F., Gömez, E., & De Prato, G. (2021). AI watch. Defining artificial intel- 
ligence 2.0. Towards an operational definition and taxonomy for the AI landscape. Publica- 
tions Office of the European Union. 

Goralski, M. A., & Tan, T. K. (2020). Artificial intelligence and sustainable develop- 
ment. The International Journal of Management Education, 18(1), 100330. https://doi. 
org/10.1016/j.ijme.2019.100330. 

Hauff, M. (2021). Nachhaltige Entwicklung. De Gruyter. 

Herdmann, F. (2023). Resilienz in Unternehmen. Beuth DIN. https://www.beuth.de/de/ 
themenseiten/resilienz-in-unternehmen. Zugegriffen: 1. Sept. 2023. 

Holzbaur, U. (2020). Nachhaltige Entwicklung — Sustainable Development. In U., Holzbaur 
(Hrsg.), Nachhaltige Entwicklung (S. 1-41). Springer Fachmedien Wiesbaden. 

ISO (2017) ISO 22316:2017. Security and resilience—Organizational resilience—Princi- 
ples and attributes. ISO. https://www.iso.org/standard/50053.html. Zugegriffen: 1. Sept. 
2023. 


18 


22. 


23. 


25. 


25. 


26. 


27. 


28. 


29. 


30. 


31. 


32. 


33. 


C. Z. Wehlmann 


Ivanov, D. (2020). Predicting the impacts of epidemic outbreaks on global supply chains: 
A simulation-based analysis on the coronavirus outbreak (COVID-19/SARS-CoV-2) case. 
Transportation Research Part E: Logistics and Transportation Review, 136, 101922. https:// 
doi.org/10.1016/j.tre.2020.101922. 

Jeronen, E. (2013). Sustainability and sustainable development. In S. O., Idowu, N. Ca- 
paldi, L., Zu, A. D., Gupta (Hrsg.), Encyclopedia of corporate social responsibility, 
(S. 2370-2378). Springer Berlin Heidelberg. 

Klasen, S. (2002). Low schooling for girls, slower growth for all? Cross-country evidence 
on the effect of gender inequality in education on economic development. The world bank 
economic review, 16(3), 345-373. 

Modica, M., & Reggiani, A. (2015). Spatial economic resilience: overview and perspecti- 
ves. Networks and Spatial Economics, 15(2015), 211-233. https://doi.org/10.1007/s11067- 
014-9261-7. 

Rohde, F., Wagner, J., Reinhard, P., Petschow, U., Meyer, A., Voß, M., & Mollen, A. (2021). 
Nachhaltigkeitskriterien für künstliche Intelligenz, Bd 220. 

Seeliger, L., & Turok, I. (2013). Towards sustainable cities: Extending resilience with in- 
sights from vulnerability and transition theory. Sustainability, 5(5), 2108-2128. https://doi. 
org/10.3390/su5052108. 

Vinuesa, R., Azizpour, H., Leite, I., Balaam, M., Dignum, V., Domisch, S., Felländer, A., 
Langhans, S. D., Tegmark, M., & Fuso Nerini, F. (2020). The role of artificial intelligence 
in achieving the Sustainable Development Goals. Nature communications, 11(1), 233. 
https://doi.org/10.1038/s41467-019-14108-y. 

Vohra, S., Vasal, A., Roussiere, P., & Guan, L. (2022). The art of AI maturity: Advancing 
from practice to performance. Accenture Insights. 

Wittenbrink, N., Demirci, S., & Wischmann, S. (2023). Resiliente und robuste KI-Systeme 
im praktischen Einsatz. In V., Wittpahl (Hrsg.), Resilienz (S. 199-211). Springer Berlin 
Heidelberg. 

Yossi, S. (2005). The resilient enterprise: overcoming vulnerability for competitive advan- 
tage. The MIT Press. Pearson Education. 

Zamani, E. D., Smyth, C., Gupta, S., & Dennehy, D. (2022). Artificial intelligence and big 
data analytics for supply chain resilience: A systematic literature review. Annals of Opera- 
tions Research, 1-28. https://doi.org/10.1007/s10479-022-04983-y. 

Zinke-Wehlmann, C., Friedrich, J., Kirschenbaum, A., Wölke, M., & Brückner, A. (2022). 
Conceptualizing sustainable artificial intelligence development. In L. M., Camarinha-Ma- 
tos, A., Ortiz, X., Boucher, A. L., Osörio (Hrsg.), Collaborative Networks in Digitalization 
and Society 5.0 (Bd. 662, S. 545-554). Springer International Publishing. 


Resilient and Sustainable Al. Positioning paper on the relation ... 19 


Open Access Dieses Kapitel wird unter der Creative Commons Namensnennung 4.0 Inter- 
national Lizenz (http://creativecommons.org/licenses/by/4.0/deed.de) veröffentlicht, welche die 
Nutzung, Vervielfältigung, Bearbeitung, Verbreitung und Wiedergabe in jeglichem Medium und 
Format erlaubt, sofern Sie den/die ursprünglichen Autor(en) und die Quelle ordnungsgemäß 
nennen, einen Link zur Creative Commons Lizenz beifügen und angeben, ob Änderungen vor- 
genommen wurden. 

Die in diesem Kapitel enthaltenen Bilder und sonstiges Drittmaterial unterliegen ebenfalls 
der genannten Creative Commons Lizenz, sofern sich aus der Abbildungslegende nichts ande- 
res ergibt. Sofern das betreffende Material nicht unter der genannten Creative Commons Lizenz 
steht und die betreffende Handlung nicht nach gesetzlichen Vorschriften erlaubt ist, ist für die 
oben aufgeführten Weiterverwendungen des Materials die Einwilligung des jeweiligen Rechte- 
inhabers einzuholen. 


Digital transformation of organizations 
or processes through AI 
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Abstract. This paper presents the development of a human-centred AI 
system for the classification of laundry according to washing categories 
such as color and type. The system aims to provide a solution that is both 
accurate and easy to use for individuals with varying levels of technical 
expertise. The development process involved a human-centred approach, 
including user research and testing, to ensure that the system meets the 
needs and expectations of its users. The system uses a combination of 
computer vision techniques and machine learning algorithms to analyze 
images of dirty laundry and provide recommendations for the appropriate 
washing category. 

In addition to the development of the system itself, this paper also 
focuses on the explanation of the AI. The aim is to increase transparency 
and promote understanding of how the system makes its decisions. This 
is achieved through the use of visualizations and explanations that make 
the inner workings of the AI more accessible to users. 

The results of testing demonstrate that the system is effective in 
accurately classifying dirty laundry. The explanation of the AI has yet 
to receive more feedback, whether users report that it increased their 
trust in the system and find it easy to use. The development of a 
human-centered AI system for laundry classification has the potential 
to improve the efficiency and accuracy of laundry sorting while also pro- 
moting understanding and trust in AI systems. 


Zusammenfassung. In diesem Beitrag wird die Entwicklung eines men- 
schenzentrierten KI-Systems für die Klassifizierung von Wäsche nach 
Waschkategorien wie Farbe und Typ vorgestellt. Das System zielt darauf 
ab, eine Lsung zu bieten, die sowohl einfach wie auch möglichst genau 
fiir Personen mit unterschiedlichem technischem Fachwissen zu bedienen 
sein soll. 

Das System nutzt eine Kombination aus Computer-Vision- 
Techniken und Algorithmen des Deep Learning, um Bilder von 
schmutziger Wasche zu analysieren und Empfehlungen fiir die richtige 
Waschkategorie zu geben. Neben der Entwicklung des Systems selbst geht 
es in diesem Beitrag auch um die Erklärung der KI und das Aktive Ler- 
nen. Ziel ist es, die Transparenz zu erhöhen und das Verstndnis dafür zu 
frdern, wie das System seine Entscheidungen trifft. Dies wird durch den 
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Einsatz von Visualisierungen und Erklärungen erreicht, die den Nutzern 
die Funktionsweise der KI näher bringen. Durch das Aktive Lernen wird 
der Aufwand fr das Annotierten der Daten verringert, welches für jede 
Wscherei aufgrund unterschiedlicher Bedürfnisse erneut durchgeführt 
werden müsste. 

Die Testergebnisse zeigen, dass das System in der Lage ist, bes- 
timmte Attribute schmutziger Wäsche zuverlässig zu klassifizieren. Es 
sind zuknftig Nutzerstudien notwendig, welche überprfen, ob das Sytem 
tatschlich das Vertrauen in das System strkt und es einfach zu bedi- 
enen ist. Die Entwicklung eines menschenzentrierten KI-Systems zur 
Wascheklassifizierung hat das Potenzial, die Effizienz und Genauigkeit 
der Wäschesortierung zu verbessern und gleichzeitig das Verstndnis und 
Vertrauen in KI-Systeme zu fördern. 


Keywords: Computer Vision * Deep Learning - XAI - Human-centred 
approaches of AI design 


1 Introduction 


1.1 Problem 


In industrial laundry technology, the sorting of dirty laundry according to wash- 
ing categories has so far either not been carried out at all, has been done by scan- 
ning barcodes and RFID chips, or has been carried out manually by employees 
under safety measures. The latter requires human contact with the soiled and 
often contaminated laundry, which represents a great physical and psychologi- 
cal burden for the employees and also involves a considerable health risk. This 
problem arises in almost all industrial laundries (several hundred companies in 
Germany), so that support for laundries in this problem area would mean a huge 
improvement in working conditions for thousands and thousands of employees. 

However, the application of Al in the laundry industry is currently very prob- 
lematic. As in other industries, the term AI has extremely negative connotations 
because there is a general fear that AI will destroy jobs. Employees and their rep- 
resentatives are therefore skeptical about the introduction of AI. This is largely 
justified and must be part of a company’s AI introduction strategy. 

In some areas, however, the skepticism is also based on ignorance and mis- 
conceptions about AI and its practical applications. The actual goal of and the 
actual changes brought about by Al in the work process must be made clear. 
Educating employees through further training is thus a central prerequisite for 
the successful introduction of Al. 


1.2 Objective and Solution Approach 


The objective is to minimize the contact of working people with soiled and 
possibly contaminated laundry by automatically classifying the laundry delivered 
to the laundries so that it can be treated without direct human contact. 
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Based on an existing automatic machine for laundry separation, a solution 
for laundry sorting according to washing programs, primarily based on camera 
images and AI, is to be developed. 

In addition, it must be taken into account that different types of laundry 
are washed in different laundries, so that a human-machine interaction module 
for intuitive training of the concrete requirements of a specific laundry shop will 
also be co-developed using “Active Learning” methods. 

Moreover, the learned knowledge of the trained artificial neural network is 
not to function as a “black box”, but rather current approaches of “Explainable 
Artificial Intelligence” are to be adapted and further developed to meet the needs 
of employees in medium-sized companies as well as increase their acceptance of 
Al. 

This creates a new division of labor between humans and Al-controlled 
machines. This process of introducing AI control is to be supported by a qual- 
ification process that prepares and trains employees who are unaccustomed to 
cooperate with an AI system. The focus here is not only on technical training, 
but also on conveying a basic understanding of how this AI system works and 
the knowledge acquired, in order to minimize reservations about the use of this 
Al-based system. 

Summarizing, the addition of a classification component to a wash chain of 
an industrial laundry shop improves working conditions in the area of laundry 
sorting. This is beneficial for the employees in industrial laundries and makes 
it easier for the mostly medium-sized companies to recruit workers for this not 
particularly popular area of work. 


2 Related Work 


2.1 Developing a Human-centred AI in an Industrial Setting 


By taking over routine tasks (such as information search), the workload of 
humans (e.g., machine operators) can be reduced and value creation increased 
by focusing on core tasks. In the form of intelligent assistance systems, AI appli- 
cations can support humans in specific tasks. To do this, the assistance systems 
analyze the current situation and make predictions. They are context-sensitive 
and can successively interact better with humans in a natural way (e.g., under- 
standing speech) [11]. However, it is important to note that production condi- 
tions are subject to temporal variations, such as changing workloads and aging 
sensor systems. The data generated from this is not stationary, it is subject 
to drift, i.e. the observed data distribution is fundamentally different from the 
data used to train the systems. Efficient adaptive fusion architectures, where 
the learning process can be controlled by detected drift, are needed to facilitate 
human decision making in such problems while reducing complexity [4,5]. 
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Furthermore, humans can adapt the AI, to minimize the drift. In order to 
accomplish this, however, it is necessary to make the AI’s decisions understand- 
able even to people with low technical expertise. In order to break the black-box 
character of modern AI models, a number of possibilities have been developed 
in recent years under the keyword “Explainable Artificial Intelligence” (XAT) to 
visualize the learned models of neural networks and to make them interpretable 
[12,17]. Thus, various methods of active learning (AL) have been developed for 
these networks, which, on the one hand, actively use human expertise in training 
these networks, and thus also reduce the necessary amount of hand annotation 
of huge datasets, which is particularly daunting for small- and medium-sized 
companies [13,16]. Through these methods, it is also possible to increase the 
understanding and therefore also trust of users. 


2.2 Laundry Classification 


Classification of textiles and cloth has been the subject of research in vari- 
ous fields, including computer vision, machine learning, and textile engineering. 
There are two popular open datasets, FashionMNIST and DeepFashion, both 
focusing on classifying the type of the clothing object. The pictures show the 
garments frontally and not bent/creased, thus in a totally different setting with 
less complexity compared to this project. 

Due to the global increase in textile waste, various governments have issued 
orders for the recycling of textiles [19,20] and [21], increasing the research of clas- 
sification of fabric. Kampouris et al. [10] used geometry to classify fabrics based 
on their surface textures and reflectance, while Sonawane et al. [15] employed 
a convolutional neural network (CNN) to recognize different types of clothing 
fabrics. Additionally, recent advances were able to achieve good classification 
rates based on hyperspectral data [3]. 


3 Methods 


In this chapter, we will first discuss the conception of the project. Afterwards, 
the AI methods and those that support and simplify the work with the AI will 
be described. 


3.1 Experimental Setup (AreaScan) 


At first, a system was designed for a test bed, which is to be used for the 
recognition and classification of individual pieces of laundry. The concept can 
be seen in Fig. 1. 
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Fig. 1. Design of the identification process. The recognition module is shown on the 
left. When a piece of laundry passes over the conveyor belt, an image is stored. This 
is then processed by neural networks, which predict the individual attributes of the 
laundry item (right) to decide upon the correct washing category 


In the middle of this module there is a gray conveyor belt over which the 
laundry pieces travel through the module one by one. The speed of the conveyor 
belt is 0.6m/s. A FRAMOS D415e depth camera with a resolution of 1280 X 720 
pixels is mounted above the conveyor belt. In order to filter out variations in 
illumination, the conveyor belt surface is isolated from above, below and the 
sides by the walls and the conveyor belt itself. Lighting is provided by two LED 
strips in front of and behind the camera. 

For the recording, the laundry pieces are separated and dropped at three 
different positions onto the conveyor belt. This is to simulate the dropping of a 
gripper arm, which in the future will take over the separation and dropping of 
the laundry pieces. The laundry then runs along the conveyor belt and reaches 
the camera’s pickup area. When the camera detects a piece of laundry, several 
images are taken of the piece of laundry at regular intervals. This is necessary 
because some laundry items are too large for the recording area. A disadvantage 
of this methodology is that the laundry item is photographed from only one side 
and may also be scrunched up due to dropping. However, characteristics such as 
the color, type, and material should still be recognizable. In the case of soiling 
that is only visible on the underside, however, it is not possible to photograph 
it from above. Accordingly, the soiling would not be visible on the pictures. 
However, this restriction had to be made because in many laundries there is not 
enough space to hang up the laundry, so that it may be photographed from both 
sides. A possible solution to this problem could be a mechanism that turns the 
piece of laundry to the other side, whereupon another picture is taken. 

In the next step, a manual annotation of the relevant classes is done. These 
categories are: color (white, black, red tones, blue tones, ...), type (shirt, trousers, 
gown, ...), degree of contamination (none, strongly, infectious, ...), kind of soiling 
(ink, mildew, blood), damage (none, chemical, mechanical) and washing temper- 
ature (30°, 60° and 90 (75)°). The classes are selected very finely. An adjustment 
for coarser classes for sorting is thereby also easily possible. 


Adapted Setup (LineScan) When the module was exhibited and tested at a 
different location, the segmentation, and therefore the classification, was some- 
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times massively affected due to different lighting conditions despite the measures. 
Thus, the explanations and the observations we noticed during the annotation 
and segmentation process serve as a basis to make some optimizations to the 
recognition module. 

The goal is to eliminate both the interfering objects and variance due to e.g. 
different lighting conditions. The interior should be shielded from the outside 
as much as possible, have a homogeneous/same color background and be well 
illuminated. Since we have laundry pieces of different lengths and depth data 
did not improve the classification, we decided to use a line scan camera as a 
reasonable adaptation. The line of the camera is aligned in such a way that it 
does not contain any interfering objects such as black blocks or holes. In addition, 
it is then only necessary to paint the walls in the area of the line to match the 
conveyor belt color and to adjust the illumination. Using this setup and using 
presorted laundry, two people were able to take and annotate 5,430 pictures over 
6 days, which is a speedup of factor 4.3 when considering the number of laundry 
items passing the camera compared to the first setup. 

Due to feedback on our first working demonstrator and discussions with 
experts in the washing industry over time, we also adapted the categories. Color 
as well as soiling were extended by a few classes as well as considered as a multi- 
label classification problem. For the type, we only added a few new classes. We 
also added a new class “washing color”, describing the color group with which 
the laundry item is to be washed together. 


3.2 AI in Image Processing 


We focus on the use of convolutional neural networks (CNN) [7], which are nowa- 
days used in almost all areas of computer vision due to their great performance. 
These are artificial neural networks, which are particularly suitable for image 
data due to their design. 

In order to simplify the handling of these methods for the employees, we 
also deal with methods of active learning and the explainability of artificial 
intelligence. 

The way in which the neurons are arranged and connected is called the archi- 
tecture of the artificial neural network. So far in this project, we have looked at 
architectures that achieved the best performance on the ImageNet dataset at the 
time of their publication [18]. For classification, the CNN architectures VGGNet 
[14] and DenseNet [6] were initially implemented to gain first insights. The basis 
for this decision was the low complexity of the VGGNet and the parameter effi- 
ciency of the DenseNet. For each of these architectures, one network per category 
was trained and optimized to keep the complexity as low as possible initially. 

As we identified that the networks learned incorrect shortcuts for the classifi- 
cation (e.g. it has learned, that if the black blocks were visible at the edge, it has 
tended to be a small piece of laundry, which in almost all cases were colorful), 
we also trained PerturbGAN [1] and a U-Net [8] to extract the laundry object 
from the background. Therefore, only the area of the piece of laundry can be 
used for the classification. 
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3.3 Active Learning 


The required annotation of this data adds to the effort, which would also have 
to be redone specifically for each different laundry shop due to different cus- 
tomers. Therefore, it makes sense to develop strategies that reduce the num- 
ber of required annotated data or simplify the annotation. Such algorithms are 
assigned to the field of active learning. 

The authors of the CEAL algorithm [16] (cost-effective active learning) pro- 
pose to adopt the certain classifications obtained by the neural network on unla- 
beled data and use them in addition to the already annotated data for a new 
training of the same AI. This process is illustrated in Fig. 2. This process is 
repeated until either a satisfactory recognition performance of the neural net- 
work is achieved or the complete dataset is annotated. Such human-in-the-loop 
concepts can also increase acceptance and understanding. 


3.4 Explainable AI 


As already mentioned, the decision path of deep neural networks is difficult to 
understand. Artificial neural networks usually contain millions of parameters 
and links, which makes it very difficult for a human to keep track of them. This 
is why CNNs are also called black-box models. Explainable AI (XAI) methods 
try to shed light into this black box so that the neural network’s decisions can 
be understood. In this project, we first looked at layer-wise relevance propa- 
gation (LRP) [2], as it produces an easy to understand heatmap in the input 
image indicating the level of importance of each pixel contributing to the final 
classification outcome. 
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Fig. 2. The CEAL paradigm gradually feeds samples from the unlabeled dataset into 
a neural network (a). Then, the selection criteria for the certain images and the most 
informative/uncertain images are applied to the classification results of the neural 
network (b). After adding user-annotated uncertain images to the annotated dataset 
(c) and the pseudo-annotations of non-user-annotated certain images (d), the model is 
further updated/trained (e). Adapted from Wang et al. [16] 


LRP considers the inputs and outputs of the network. Intuitively, it uses the 
network weights and the activations of the neurons generated by the forward 
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pass to propagate the output back through the network to the input layer. This 
gives the input layer the relevance of each pixel to the predicted class. From 
this it can be deduced which pixels the network uses as positive (red), slightly 
positive (yellow) or neutral (green) indicators for the respective classification. 
Some LRP explanations in the context of this project can be seen in Fig. 3. 


4 Results 


We created two datasets, consisting of 9,405 (AreaScan) and 5,430 (LineScan) 
images. For each of the images, information exists about the color, the type, and 
mostly also about the soiling, the damage, and the material. For the AreaScan 
dataset, segmentation was considered before classification, so we present these 
results first below. 


4.1 Segmentation 


First, 5,000 of the images were segmented by hand, as the unsupervised Perturb- 
GAN did not provide good results. Subsequently, we iteratively trained a U-Net, 
predicted new segmentations for the remaining images and either accepted or 
rejected them until all images were segmented. Through this active learning ori- 
ented process, 2,091 segmentations were automatically created. Without deduct- 
ing the effort for accepting/rejecting, this saved about 22.23% (here specifically 
32 hours) of work time. The segmentation results can be seen in Tab. 1. 


Table 1. Accuracy and IoU on the test dataset of the best model for the AreaScan 
dataset. For comparison, we calculated the segmentation by means of difference to an 
image of an empty conveyor belt and the output of the GrabCut [9] algorithm without 
corrections 


Score Difference GrabCut PerturbGAN U-Net 
Accuracy 0.8971 0.9425 0.8864 0.9993 
IoU 0.6989 0.7994 0.6106 0.9947 


4.2 Classification 


Dataset AreaScan In this dataset 6,855 images (72.89%) are of laundry items 
that originate from nursing homes and 2,550 images (27.11%) are of workwear. 
Since only for the color, the type and the soiling to more than 90% the classes 
could be determined with certainty and the dataset is already rather small in 
general, the focus was initially placed on these three categories. 
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For both CNN architectures, hyperparameter optimization was performed 
for each category combined with each dataset (unprocessed and segmented). 
We first restricted ourselves to the learning rate, batch size and to the use of 
a pre-trained network. Here, the best scores on the test dataset are listed in 
Tab. 2. 


Table 2. Accuracy on the test dataset of the best model for the AreaScan dataset 


Dataset Color Type Soiling 
AreaScan 0.9649 0.7989 0.7841 
AreaScan (Segmentations) 0.9539 0.7952 0.8100 


Dataset LineScan Here, using the 5,430 images, we also performed a hyper- 
parameter optimization for both CNN architectures for each category. Since we 
now considered the classification of color as well as soiling from the creation of 
this dataset as a multi-label problem, we observed the F} score for these classes 
and had to change to a sigmoid activation function for the output. Furthermore, 
height equalization as preprocessing and data augmentation strategies like ran- 
dom rotation and vertical/horizontal flipping during training were performed. 
Here, the best scores on the test dataset are listed in Tab. 3. 


Table 3. Accuracy (type, washing color) and Fı score (color, soiling, damage) on the 
test dataset of the best model for the LineScan dataset 


Dataset Color Type Soiling Washing Color Damage 
LineScan 0.8230 0.9652 0.5783 0.9713 0.8537 


4.3 Explainable AI 


In addition, the LRP algorithm has been implemented and can be applied to 
any network architecture. A visualization of the color classification explanations 
along with the certainty of the respective neural network can be seen in Fig. 3. 
Looking at the classification with high certainty (a-b, > 98%), it is noticeable 
that only regions of the laundry piece were used for the classification. In the less 
certain range (c, 80 — 90%), the laundry piece was used for the most part, but 
also areas at the edge. In a relatively uncertain classification (d, < 70%), the 
laundry piece was obviously not recognized and only edge pixels and pixels at 
the transition between conveyor belt and wall were used. 
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Fig. 3. Original AreaScan pictures (left) and the color classification including the LRP 


explanation (right). The relevance of each pixel for the classification (above) is color 
coded (green — neutral, red — very important) 


4.4 Output of the System 


Finally, Fig. 4 shows the representation of the whole output including certainties 
for each class and visualizations for a randomly chosen piece of laundry. 


Input Segmentation Color Type Soiling 


Eating apron: 0.0081 
Apron: 0.0052 
Incontinence pad: 0.0051 
Sweatshirt: 0.0046 


Pigment: 0.0290 
Usage stain: 0.0184 
Mildew: 0.0112 
Stool: 0.0082 


‘White: 0.9607 
Colorful: 0.0393 


Fig. 4. Original images with the segmentations of the trained U-Net and the outputs 
of the most probable classes of a trained neural network per category. Here, the most 
probable class is highlighted in yellow. In addition, the relevance of the individual input 
pixels for the classification of the neural network is calculated using LRP and displayed 
in the form of a heatmap (green — neutral, red — very important) 


5 Discussion 


The CNNs were able to achieve good classification accuracies for all categories, 
although the AreaScan dataset being fairly small. By using LRP, unwanted 
strategies could already be detected, which is why networks were trained for 
segmentation. Since the unsupervised segmentation approach did not give good 
results, segmentations had to be created manually. Through the successful use 
of CEAL for creating the segmentations, time for the annotation process could 
be saved. The subsequent segmentation performance of the U-Net is very good. 
However, the classification could not be improved by using the segmented images. 
Nevertheless, the use of the shortcut could be eliminated. 
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Since, on the one hand, the laundry items were not completely visible on 
the images at the time and, on the other hand, tests revealed some optimization 
possibilities, the setup was adapted using a line scan camera, among other things. 
In addition, the recording process was further optimized, which quickly led to 
the creation of a second larger dataset (LineScan). On this one, the classification 
performances of the first trainings are for some classes already very good. In the 
next steps, CEAL will be used to further increase the size of the LineScan dataset 
and to continue training models. In parallel, further user studies are planned. 


6 Conclusion 


The paper proposes a human-centred system for sorting laundry based on deep 
learning models. The system uses convolutional neural networks to classify laun- 
dry items based on their visual features, and layer-wise relevance propagation to 
explain the model’s predictions in a human-understandable way. The proposed 
system offers a promising approach for developing a user-friendly laundry sort- 
ing system which aims to take human needs and preferences into account, which 
will be examined in a subsequent study. 
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Abstract. With the increasing prevalence of mobile applications across various 
domains, there is a growing demand for individualised and self-adaptive lear- 
ning pathways. This is particularly important in the mobile health sector, where 
there is a critical need to investigate how expert and experiential knowledge can 
be acquired, digitalised and formalised into data which is subsequently proces- 
sed and further used. To address this demand, our research explores how Artifi- 
cial Intelligence (AI) can power this process. We developed a prototype mobile 
application with a standardised learning pathway that features speech-language 
therapy exercises of varying levels of difficulty. In a 12-week field experiment 
involving 21 individuals with aphasia, we analysed the results using supervised 
and unsupervised algorithms. Our findings suggest that AI has the potential to 
generate new knowledge, such as identifying features that can determine which 
learning words are perceived as easier or more difficult on an inter-individual 
basis. This knowledge enables algorithmisation and the design of standardised 
(database-supported) artefacts, which in turn can be used to formulate self-ad- 
aptive and individualised learning pathways. This significantly enhances the 
development of effective mobile applications to assist speech-language therapy. 


Keywords: Artificial Intelligence - Expertise Mining - Mobile Health - 
Speech-Language Therapy 


1 Introduction 


As the world becomes increasingly digitalised, there is a growing emphasis on the 
human element within technological environments. Technology is being used to adapt 
processes in e.g., production, healthcare, and education to better serve human needs 
[1]. With the shift from Industry 4.0 to Industry 5.0, the focus is set on promoting 
human well-being and developing skills and abilities [2]. Artificial Intelligence (AI) 
is an important and rapidly evolving technology that can support this process. Quali- 
fication processes for humans are of particular interest and relevance, for example, in 
training new employees and reskilling workers in the context of work, as well as in re- 


© Der/die Autor(en) 2024 

C. Zinke-Wehlmann und J. Friedrich (Hrsg.): First Working Conference on Artificial 
Intelligence Development for a Resilient and Sustainable Tomorrow, AIDRST 2023, 
Informatik aktuell, S. 37-49, 2024. https://doi.org/10.1007/978-3-658-43705-3_4 


38 J. Kauper et al. 


covery from serious illnesses or medical procedures. In all these cases, there is a need 
for individualised learning processes that are tailored to each individual’s skill level 
and any relevant restrictions (e.g., health or knowledge). Our goal is to formulate in- 
dividualised learning content and exercises to optimise the learning progress for each 
participant. In this paper, we investigate how AI can support this process by providing 
deeper insights into the available data base. The paper is structured as follows: Sect. 2 
provides background information, Sects. 3 presents the proposed methods, Sect. 4 and 
5 describe the experimental results, and Sect. 6 provides a conclusion and outlook for 
future work. 


2 Theoretical Background and Current State of the Art 


AI as a branch of computer science deals with the development of algorithms to per- 
form tasks traditionally associated with human intelligence, like the ability to learn 
and solve problems. A thorough statistical basis and an introduction into Machine Le- 
arning approaches is given in [3]. In [4], the focus is set on the algorithmic view- 
point and e.g. complexity analysis, whereas [5] provides an introduction into Deep 
Learning. AI applications are already prevalent in diagnostics, especially in the field 
of computer vision for tumour identification, see e.g., the study [6]. Recent develop- 
ments were achieved in disease classification based on segmentation [7] and in natural 
language processing [8]. 

Especially in the context of Industry 5.0, the focus is set on the “human in-the- 
loop” (HITL), whereas Al is seen as supporting technology: The human provides an 
extensive data base of high quality (which is required to not be biased), detects poor 
training results, interprets them and extracts knowledge for further use. Explainable 
AI for methods which do not provide such an explanation of the solution-finding pro- 
cess (as e.g. the decision tree algorithm does) is an avid field of research. For an over- 
view, see [9]. However, the results are mostly theoretical and not yet in application. 
The choice of an algorithm highly depends on the planned usage of the results. From 
a practical point of view, users prefer to apply algorithms that provide a high level 
of (inherent) explainability. Measuring learning progress and learning analytics is an 
active field of research, as evidenced by numerous publications. A reference model is 
given by [10], whereas [11] provides a framework of quality indicators to standardise 
the evaluation of learning analytics tools. Various empirical studies and implementati- 
ons, such as [12], also exist. 

HITL and Al-enabled adaptive learning systems have several capabilities such as 
generating individualised learning pathways adapted to individual progress, preferen- 
ces, and pace [13]. A user-centred approach enables system feedback and correction, 
adapting to users’ learning styles and knowledge, providing individualised learning 
pathways aligned with specific needs and individual goals [14], ensuring higher ac- 
curacy and effectiveness by adapting to changing conditions and data [13]. Adaptive 
learning systems have broad applications across domains, including language educa- 
tion using frameworks such as the Common European Framework of Reference for 
Languages (CEFR) [15]. Speech-language therapy lacks such frameworks and relies 
on the expert and experiential knowledge of speech-language therapists (SLTs) using 
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different treatment guidelines, techniques, and technologies based on the principles 
of evidence-based practice (EBP), comprehensive diagnosis, and individual therapy 
goals [16]. 

Digital applications used in speech-language therapy include high-tech augmen- 
tative and alternative communication (AAC) tools such as speech-generating devices, 
apps on tablets [17] and interactive tables with touchscreens [18]. There is also an 
increasing usage of mobile apps [19]. Machine learning can enable SLTs to automati- 
cally adjust training and rehabilitation devices remotely based on a tailored treatment 
plan [20], thereby providing a personalised and more effective therapy experience. 
Hence, there is a growing need to explore expert and experiential knowledge in data 
to design self-adaptive and individualised learning pathways. 


3 Methodology 


To create self-adaptive and individualised learning pathways for mobile applications 
in speech-language therapy, expert and experiential knowledge needs to be translated 
into data and implemented as artefacts in an app for testing and evaluation. Following 
an iterative approach, we combine two different methods to achieve this goal: 1) the 
enhanced Action Design Research (ADR) process for the design and development of 
the mobile app, and 2) the Cross-Industry Standard Process for Data Mining (CRISP- 
DM) as part of the evaluation to gain domain knowledge and thus improve the arte- 
facts. 

ADR aims to tackle a real-world problem within a specific organisational con- 
text while simultaneously designing and evaluating an IT artefact that deals with a set 
of issues inherent to that situation [21]. By incorporating a field experiment into the 
ADR process for iterative refinement and evaluation of the mobile app, we combined 
design science and field experiments as described in [22]. 

CRISP-DM is an open, industry-, software- and application-independent standard 
process model and provides an approach for data mining processes. The model con- 
sists of six phases: Business Understanding, Data Understanding, Data Preparation, 
Modelling, Assessment and Deployment. [23] 

The following sections describe the application and results of these methods in 
more detail. 


4 Enhanced ADR and Field Experiment 


The app development and the associated transformation of expert and experiential 
knowledge into data were guided by the enhanced ADR with the following stages: 
Stage 1) Problem formulation: To understand how a mobile app can support 
speech-language therapy, patients’ specific needs and objectives were identified fol- 
lowing EBP as recommended by the professional guidelines for speech therapy of the 
German Federal Association for Speech Therapy (dbl) [24], taking into account fac- 
tors such as patient expectations, therapist’s clinical experience, and available scien- 
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tific evidence [25]. Data was collected through scientific research and interviews 
with SLTs to investigate feasibility, including information on patient diagnoses and 
individual therapy goals. Therapy goals take into account factors like speech impair- 
ment type and severity, individual needs and preferences, and participation limitati- 
ons according to the International Classification of Functioning, Disability and Health 
ICF [26]. Despite challenges in achieving standardisation, considerations were made 
towards standardisation and algorithmisation in designing a standardised database- 
supported portfolio that considers individual needs, the severity and manifestation of 
speech disorders, and different therapy goals. 

Stage 2) Building, intervention, and evaluation: SLTs designed a uniform core le- 
arning pathway for all participants as a foundation for creating individualised lear- 
ning pathways based on therapy goals, diagnoses, patient-specific needs and abilities, 
and app usage data. This core learning pathway consisted of a predefined sequence of 
three evidence-based speech-language exercises per day on at least five days per week 
such as picture-word association, spelling, and selection of the correct sentence with 
increasing difficulty. Semantic fields (categories) with medium to high intersubjec- 
tive relevance to daily life of the participants, including food, clothing, housing, body, 
everyday objects, garden, illness, and personal care, taking into account the gender 
and age distribution of the patients, were identified. Subsequently, we selected related 
target words, defined data requirements for the prototypical mobile app for speech- 
language therapy, and evaluated user acceptance, design, as well as usability through 
field trials. 

Stage 3) Field experiment: A 12-week field experiment was conducted from Ja- 
nuary to April 2022 with 21 post-stroke aphasia patients in Germany with varying le- 
vels of severity to assess the efficacy and usability of the app under real-world con- 
ditions. Post-stroke aphasia is a language disorder, affecting any aspect including 
speaking, writing, reading, and comprehension, that arises following a stroke [27] 
and is often accompanied by comorbidities such as motor impairments like hemipa- 
resis and cognitive deficits like memory loss [28]. Eligibility criteria included a dia- 
gnosis of aphasia, premorbid competence in German, and sufficient auditory/visual 
acuity to interact with a tablet. Some eligible participants were excluded due to re- 
source constraints, i.e., the lack of access to a device or internet. Participants’ mean 
age was 63.6 years (range 38-81), and 67% were male. Personal information was pro- 
vided through an anamnesis questionnaire, including visual and hearing impairment, 
handedness, and stroke history. Data on the specific type of aphasia, severity, type of 
stroke and co-morbidities were not collected as they were not considered relevant to 
the inclusion or exclusion criteria. This is because the design of the speech-language 
exercises aimed to take into account the heterogeneity of clinical disorders observed 
in people with aphasia. User feedback was collected through two questionnaires: a 
usability test at first use and a final questionnaire, which included self-assessment of 
the difficulty of the exercises. Progress was monitored, analysed, and evaluated at 
both individual and inter-individual level. 

Stage 4) Reflection and learning: Through the evaluation of the field experiment’s 
results, we engaged in deliberate and conscious reflection on the problem formulation, 
our working hypotheses, and the app artefacts that were developed. The next section 
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presents and discusses the outcomes of the first five phases of the CRISP-DM deploy- 
ment process. These results will be applied in Stage 5 of the enhanced ADR, Formal- 
isation of learning, to optimise and formalise our models and assumptions. 


5 Exploration of Expert Knowledge Using CRISP-DM 


To optimise the learning curve for each individual patient, the exercises have to con- 
tain words of a suitable level of difficulty. In the following, data-based evidence is 
extracted to evaluate the chosen level of difficulty for each word and to provide re- 
levant features, extract potentially irrelevant features and suggest additional features 
to be acquired. Therefore, we applied CRISP-DM to the example of the picture-word 
association exercise and its difficulty factors to answer the following two questions: 
(1) How do the assumptions on difficulty-determining features affect the frequency 
of errors per word across users with heterogeneous clinical presentations in terms of 
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“easy”, “medium” or “hard”? (2) Which features lead to the classification of a target 
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word as “easy”, “medium” or “hard”? 


5.1 Understanding of the Use Case 


The picture-word association exercise in the app prototype presents a picture of the 
target word on the right side of the screen and four written words (the target word and 
three distractors) on the left and centre. Participants must correctly identify and match 
the word with the corresponding picture. The exercise’s difficulty increased weekly 
based on factors such as the type of exercise, prototypicality and unambiguous repre- 
sentation of the target word in the picture, semantic field (category), and, more detai- 
led below, target word/item and distractors. 

Several factors determine the difficulty of the target word. These include indi- 
vidual frequency of use [29], degree of concreteness according to the Dual Coding 
Theory (DCT) [30], prototypicality (e.g., “Möhre” (carrot Eng.) is more prototypical 
than “Schwarzwurzel” (salsify Eng.) for the category vegetables), length and struc- 
ture of the word (shorter words with simpler syllable structures (e.g., CV) are easier 
to learn than longer words with more syllables and complex syllable structures [31]), 
level of specification within a semantic hierarchy (e.g., “Hund” (dog Eng.) is easier to 
learn than “Dackel” (dachshund Eng.)), and properties such as dialectal peculiarities/ 
synonyms, imageability, number of phonemes, compound words, and foreign words. 

The properties of the three distractors (semantic, phonological, and non-relatio- 
nal) in relation to the target item also influence the difficulty level of this exercise. 
For semantic distractors, the level of difficulty increases with the closeness to the 
target item (equivalence in terms of prototypicality/type frequency, imageability/ 
visual redundancy or distinctiveness, semantic/relational proximity). For example, 
“Keks” (cookie Eng.) is an easier semantic distractor for the target word “Möhre” in 
the semantic field food than “Rettich” (radish Eng.). For phonological distractors, it 
holds that the more similar the phonological-lexical properties (compared to the tar- 
get word), the more difficult this distractor is [32]. For example, “Matratze” (mattress 
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Eng.) has the same initial letter as “Möhre”, but a different number of syllables as 
well as a different number of graphemes and is therefore an easy phonological dis- 
tractor. “Möwe” (seagull Eng.) is a more difficult distractor, as it has the same initial 
letter, same number of syllables, similar syllable structure, same vowel length, and it 
also rhymes with “Möhre”. Non-relational distractors must not be in a semantic or 
phonological-lexical relation to the target word or to any of the other distractors, like 
“Goldkette” (gold chain Eng.) for “Möhre” (carrot Eng.), as frequent errors with this 
distractor indicate guessing. 

Data collection took place during and after the 12-week test phase and included 
questionnaires, recorded app usage data, a vocabulary database with items (e.g., target 
words, and distractors) created by the SLTs, and developed attributes. The criteria that 
determine the level of difficulty of an item were elaborated and described in terms of 
their characteristics. 


5.2 Data Understanding 


This phase involved a thorough investigation of the provided database. The data on 
targets included around 25 features, with 11 having been identified as relevant for this 
investigation by the SLTs and data analysts. Hereby, certain features were neglected 
for this investigation (this mainly depended on the availability and validity of the 
data). Since the test phase was already completed, it wasn’t possible to acquire ad- 
ditional data from the test phase itself as well as patient-specific characteristics (e.g., 
displayed distractors which weren’t clicked; the patients’ familiarity with items from 
certain semantic fields). 

The selected features included categorical values like difficulty and concreteness/ 
abstractness, integer values like number of graphemes, Boolean values like com- 
pound words, and strings like structure syllables. The same attributes were provided 
for the distractors, with the additional feature “type” with its feature space {semantic, 
phonological, non-relational}. Additionally, data from the test phase was provided, in- 
cluding e.g., exercise ID, patient ID, time stamps of each click within the exercise- 
solving process by the patient, correct target value, falsely chosen distractors, used 
support (audio, joker). For a better understanding of the data, uni- and multivariate 
data analysis was performed: Histograms and boxplots allowed a deeper insight on 
distributions and possible outliers (which occurred, for example, regarding the dura- 
tion of an exercise or the total number of errors per item). Additionally, the calculation 
of correlation coefficients and category-wise variance analysis provided a deeper un- 
derstanding of the features’ connections (e.g., there is an obvious correlation between 
the length of a word and the number of its syllables, but the word type frequency is 
not correlated with the other item’s features and the type-specific number of errors). 
Appropriate visualisations, both individual and inter-individual, included e.g., the 
number of trainings per day and exercise (for each week) as stacked bar charts, the 
distribution of the types of errors (semantic, phonological, and non-relational) as pie 
charts as well as box plots, and the temporal distribution of the trainings as scatter 
plots. 
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5.3 Data Preparation 


In the context of univariate data analysis, it became apparent that the duration of some 
picture-word association exercises lasted up to 23 times as long as the average time 
and/or were not finished. Unfinished exercises were neglected; regarding the duration, 
a time limit was set in coordination with the SLTs. Furthermore, plausibility checks 
performed in Python 3.9 showed that for exercises exceeding one day, incorrect data 
was stored in the database regarding the type of exercise. These mistakes were correc- 
ted. There were no missing values. 

Further, it was required to integrate the data from the different sources. The cha- 
racteristics of the targets and distractors were provided in various Microsoft Excel 
files, as they were assembled manually by the SLTs. Data from the test phase were ac- 
quired automatically and made available in a MySQL database. The relevant data was 
extracted and structured with regard to the analysis to be performed in a new MySQL 
database. This also included the calculation of new features like the type-specific (se- 
mantic, phonological, and non-relational) number of errors per exercise. 

After data cleaning and integration, the data was then prepared in accordance with 
the AI algorithms to be applied. For the decision tree algorithm, both numerical and 
categorical data are allowed, and the data do not need to be normalised. Regarding 
clustering methods, the preprocessing process included distribution and outlier hand- 
ling, transformation and normalisation of numerical data, and removing highly corre- 
lated features. 


5.4 Modelling 


Two different AI approaches were implemented. First, the decision tree as supervised 
method was applied to identify the features and their values leading to the difficulty 
of each target (with difficulty space {easy, medium, hard}). The main advantage of 
this approach were the explainable results, which means that the features and their 
respective values leading to the categorisation could be provided and discussed with 
the SLTs. As hyperparameters, the gini criterion was chosen, with a maximum depth 
of 10 being sufficient to provide unique classification results. 

Second, clustering algorithms were applied as unsupervised methods to check 
which grouping of items was suggested. For centroid-based methods, the elbow met- 
hod to identify the optimal number of clusters led to three (which matched the number 
of categories the SLTs chose). For comparison, two and four clusters were also set as 
hyperparameters. Additionally, connectivity-based methods were applied. 

Besides the items’ characteristics (which formed the basis for the SLTs to derive the 
difficulty of the items), the results from the test phase could also be included, i.e., the 
number of mistakes for each of the three difficulty types. Hereby, the types semantic, 
phonological, and non-relational distractors were distinguished. Both approaches (lea- 
ving out and including data from the test phase) were applied and evaluated individually. 

Additionally, statistical methods for multivariate analysis were applied to compare 
the feature values of targets with same difficulty. This allowed the identification of 
conspicuous targets or peculiarities with respect to the semantic fields. For example, 
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Fig. | in Sect. 5.5 shows the comparison of selected feature values of target words 
with the same difficulty for the semantic fields clothing and food. 


5.5 Evaluation 


The results were discussed with the SLTs. Regarding the decision tree, an iterative 
investigation of the leaf nodes revealed items which weren’t categorised easily and 
required deep branching. Targets which were difficult to categorise had an 85% in- 
tersection for both approaches (leaving out and including test phase data), indicating 
a high consistency of the SLTs’ and patients’ perception. If considered, the relative 
number of errors heavily influenced the decision nodes, but not the results as much — 
especially if a sufficiently high depth was chosen as hyperparameter. Tab. 1 shows the 
impact of features selected for the algorithm with maximum depth value of 5. Here, 
the relative number of misclassified targets (also specifying the respective falsely sta- 
ted difficulty) is depicted for three cases of selected features. It becomes obvious that 
including the data acquired during the test phase with the patients significantly impro- 
ves the results. Simultaneously, the typical problem of words misclassified as “me- 
dium” is still apparent. 


Tab. 1. Number of misclassified targets under different features selected for depth 5 


Decision tree of depth 5 Easy |Medium | Hard 
Features excluding errors 

Misclassified as “Easy” = 10% 17% 
Misclassified as “Medium” |17% |- 17% 
Misclassified as “Hard” 3% 8% - 
Features including errors in total 

Misclassified as “Easy” u 12% 3% 
Misclassified as “Medium” | 3% - | 17% 
Misclassified as “Hard” 3% 2% - 
Features including all types of errors 

Misclassified as “Easy” = 2% 3% 
Misclassified as “Medium” |0% - 17% 
Misclassified as “Hard” 10% 0% - 


Five features were identified to have major impact on the categorisation. Certain 
semantic fields could be classified easily as suggested by the SLTs, whereas targets 
with “medium” difficulty were the most problematic ones. These items were further 
investigated using clustering results: Changing the number of clusters (and, hence, the 
number of difficulty categories) to two provided a deeper insight into the connection 
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between the item characteristics and the results from the test phase. Most of the target 
values with difficulty “easy” and “hard” were assigned to the two different clusters, 
respectively, indicating that the SLTs’ classification of “easy” and “hard” was mostly 
in accordance with patients’ perception. Target values with “medium” difficulty were 
reassorted. It became apparent that for each semantic field with this difficulty, either 
a reassignment to “easy” or “hard” heavily predominated. The results brought deeper 
insight into which features might be relevant for the assignment to a cluster. Creating 
four clusters is however challenging from the application point of view: A further di- 
vision of the difficulty “medium” would be necessary. As for three clusters, about half 
of the targets matched the SLTs’ classification, where the targets identified as critical 
with the decision tree method were assigned to different clusters with this method as 
well. These results in connection with the target analysis on targets with same diffi- 
culty provided the basis for the SLTs to find causes for targets with a high deviation 
from the original difficulty. Such reasons were, e.g., an unsuitable or ambiguous pic- 
ture provided in the test phase. 

Several working hypotheses were confirmed by cross-patient valid attribute rules. 
For example, the difficulty of exercises was found to depend on the type of exercise, 
and the percentage of phonological and semantic errors in the picture-word associa- 
tion exercise increased with higher difficulty, indicating appropriate distractors were 
developed for these levels of difficulty. Additionally, for the category food at an easy 
level, there was a positive correlation between word type frequency and error rate. 
This confirmed the hypothesis that words that occur more frequently in the German 
corpus (e.g., “Marmelade”, jam Eng.) would elicit fewer errors than words that occur 
less frequently (e.g., “Käse”, cheese Eng., or “Tee”, tea Eng.). However, word type 
frequency alone was not sufficient to explain the error patterns. Type frequency from 
the lexical database dlexDB [33] was used, which is based on the reference corpus of 
the German language of the 20th century from the Digital Dictionary of the German 
Language (DWDS) and ranges from 1900 to 1999. Therefore, the type frequency of 
certain words does not allow any conclusion on the current usage frequency of these 
words. For instance, we found more errors for words with shorter word length than 
for words with longer word length within the same category and level of difficulty. 
This contradicted the assumption regarding word type frequency and number of syl- 
lables. Possible factors that could account for this discrepancy are syllable structure, 
picture quality, distractor choice, or everyday relevance (depending on each patient’s 
perception). We also performed pair- and groupwise comparisons. For the two catego- 
ries food and clothing at an easy level of difficulty, the similarity between features like 
word type frequency across both categories is depicted in Fig. 1. However, there were 
fewer phonological and semantic errors for food compared to clothing. A possible 
conclusion is that the visual representation of food items may show a greater degree 
of prototypicality and a closer alignment with everyday visual perception as compared 
to items of clothing. Furthermore, food might activate other sensory modalities such 
as taste. 
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Fig. 1. Comparison of the selected feature values of target words with the same level of 
difficulty for the categories clothing and food 


6 Conclusion and Outlook 


Research on the transformation and exploration of expert and experiential knowledge 
into data using Al is crucial for the development of self-adaptive and individualised 
learning pathways in mobile applications to support patient-centred speech therapy 
treatments. In this study, we developed a prototype app, tested it with aphasic patients 
in a field experiment, and subsequently analysed and evaluated the results using diffe- 
rent AI approaches. The study confirmed some of the assumptions and criteria made 
by SLTs for developing a standardised core learning pathway, while also uncovering 
patterns and weaknesses that can enhance the revision and improvement of the proto- 
type. 

Further development of the prototype could enable patients with speech disorders 
to train continuously with increasing demands under professional supervision. The in- 
dividual therapy goal setting, taking into account participation goals (ICF) and inter- 
individual comparisons using newly developed algorithms for comparability and pat- 
tern recognition, is a pioneering basis. The significance of the current results is mainly 
restricted by the small number of patients and the brevity of the test phase. While the 
application of the decision tree algorithm allows a high transparency for the SLTs, it is 
highly susceptible to overfitting. Systematically selecting different features and apply- 
ing suitable maximum depths helped to avoid those downfalls; with a bigger data set, 
random forests as an ensemble method could also be applied. In the future, we plan to 
test Deep Learning approaches as well to get new insights. The inherent problem of 
decision trees to be biased towards the dominant class in imbalanced data sets must be 
considered as well. Clustering methods, on the other hand, as unsupervised methods, 
have the disadvantage of not providing explainable results. Additionally, the quality of 
the results highly depends on the data points’ location in the feature space, such that 
suitable algorithms must be chosen. 

To overcome the described inaccuracies, we plan to increase the sample size for 
future studies, adjust the app and vocabulary databases, and revise the criteria for de- 
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termining the difficulty of a target word and exercise. Additionally, a more precise de- 
scription of individual and inter-individual evaluations would enable a more accurate 
definition of the data collection requirements, including additional patient-specific 
data. We also recommend developing new item-specific attributes that can adapt the 
difficulty of an exercise to individual learning progress. 
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Abstract. While the concept of digital platform (-ecosystems) for the pro- 
vision of corresponding services has been met with great interest both in the 
broad re-search community and in practical application, process-based consi- 
derations for the selection of digital platform services, which are furthermore 
supported by artificial intelligence (Al), remain unexplored. However, it is pre- 
cisely the customer processes in the context of the user experience that play a 
decisive role in the success of targeted platform solutions. Therefore, this paper 
describes the development of a method that is specifically focused on the pro- 
cess-based derivation of relevant services for digital, Al-based platforms. To 
develop our method, we draw on a focus group study that operates in the en- 
vironment of a current research project for the development of an Al-based net- 
working platform and thus enables a first evaluation of the developed method. 
With our current results, we are thus not only contributing to the knowledge 
base around digital platforms and ecosystems in connection with artificial in- 
telligence but are also providing a useful action guide for developing these in 
practice. 


Keywords: Digital Services - Platform Economy - BPMN 


1 Introduction 


The design of digital services — in the sense of the platform economy of value-added 
services in addition to pure matching offers for linking customers and service provi- 
ders on transaction-centred platforms [1] — is subject to a variety of aspects. Stummeyer 
highlights three key aspects that need to be considered during their development. These 
are, firstly, usability with the focus on the product or the value-added service, secondly, 
the user experience with the focus on the customer processes and, thirdly, the custo- 
mer experience with reference to the customer of the respective provider [2]. Essential 
success factors for the user experience are utility — the subjective benefit for the target 
group — and usability — the efficient achievement of the goal by the user [2]. 
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Within the research project KISS — Al-based Rapid Supply Network, a data-cen- 
tric platform is being implemented using artificial intelligence (AI). The platform fol- 
lows the approach of an AI-based semantic market and networking platform that is in- 
tend-ed to provide further value-added services in addition to pure provider-customer 
networking [3]. The focus of the project is on linking the medical sector with manu- 
facturers from the field of additive manufacturing to increase resilience — especially in 
times of crisis — through the formation of new value creation networks. 

In accordance with the success criteria for the design of digital services mentioned 
at the beginning, the project aims to examine the extent to which the criteria for user 
experience could be systematically taken into account. The stakeholders as potential 
users of the platform, who were identified in the first step by means of a stakeholder 
analysis, are in the project-specific case employees from the medical-social sector as 
end customers and institutions in the value chain for additive manufacturing as provi- 
ders of the platform as well as others. More than 30 different actors and various rele- 
vant roles were identified. Each institution itself can be regarded as a separate entity 
with its own requirements, which needs to be mapped as broadly as possible. In order 
to map the diverse requirements, an approach was developed within the project to best 
reflect the potential benefits. As a result, we identified a research question, which is 
investigated in this paper: How should an approach be designed to derive the services 
of an AI networking platform based on processes modelled in Business Process Model 
and Notation (BPMN)? 

This article, which was developed in the project context based on a focus group 
study, emphases on the process-based definition of relevant services for Al-based se- 
mantic digital platforms, which are gaining more and more interest compared to the 
development of conventional digital ecosystems. However, how to develop Al-sup- 
ported and platform-based services has not yet been fully explored in research and 
development. Challenges are seen in the undeterministic behavior of Al-models and 
legal issues for example with the copyright for the training data. So, a one-to-one 
transfer of traditional frameworks seem not feasible. Therefore, we present an itera- 
tive approach for this purpose in order to contribute to filling this research gap. 

To this end, the remainder of the paper is structured as follows: In the next section, 
related work is introduced, and important definitions are given, before the research 
method is described in section three. Section four presents the results of the study and 
thus the developed approach. Before concluding the article with a summary and the 
presentation of some ideas for further work, possible application potentials of AI wit- 
hin the framework of the presented approach itself are also discussed. 


2 Theoretical Background 


2.1 Digital Platform Ecosystems 


Digital platforms are generally software-based artefacts that connect different stake- 
holders and facilitate interaction between them [4]. A basic distinction can be made 
between two types of digital platforms. On the one hand, there are the digital trans- 
action platforms that primarily facilitate matchmaking and transactions, such as 
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Amazon or Über [5]. Providers and consumers can thus use the platform to exchange 
goods (Amazon) or services (Über) for money. On the other hand, there are so-called 
digital innovation platforms that focus on the development of new services by provi- 
ding interfaces from the platform operator through which extensions to the platform, 
such as apps (Apple iOS) or browser extensions (Firefox), can be created [4, 5]. 

These two types are an expression of different perspectives on digital platforms, 
which are expressed in an economic perspective for transaction platforms and in an 
engineering or innovative perspective for innovation platforms [6, 7]. Whereas the 
first focusses on the mediation between two market sides [8], the latter concentrate on 
the development of the platform to offer innovative new services [5]. In this context, 
the term platform core is used, which provides basic functionalities such as user ma- 
nagement but also the tools that enable extension by third parties (for example APIs 
or other interfaces) [9]. An extended platform is created in which newly developed 
modules are provided by so-called complementarians [10]. 

Abstracted from the perspectives, digital platforms consequently consist of a plat- 
form core that provides the basic functionality and becomes an extended platform 
through complementary services. They are embedded in a (digital) eco-system that 
includes both platform components and the various stakeholders from the platform 
owner to the complementor to the customer [10]. The result is that digital platforms 
bring many actors together and enable an easy exchange of resources (e.g. digital re- 
presentations of offers) across large distances. 


2.2 Methods of Process-Centred Software Development 


For the development itself, two main streams can be distinguished: user-centred ac- 
cording to Draper and Norman and benefit-centred according to Constantine and 
Lockwood. While the focus of the former approach is on the user and the fulfilment 
of his or her needs, the latter concentrates on the improvement of the tools to system- 
ically support the tasks to be carried out [11-13]. 

The aforementioned design paradigms consider the complete scope from the first 
ideas to the development of the software. The use case described here must be distin- 
guished from this, which combines both models, but also only extends to the step of 
selecting functional elements. 

Considering the development of digital services as a system and software engi- 
neering task with a focus on requirements elicitation and management, RE for short, 
according to ISO/IEC/IEEE 29.148:2018, an iterative three-stage process with sub- 
sequent work steps can be assumed [14]: 


e Stakeholder Requirements Definition Process 
e Requirements Analysis Process 
e Architectural Design Process 


The first step is to identify the stakeholders with connections to the system over its 
life cycle, whereby according to the standard, requirements from the management 
level should also be considered in addition to pure user and developer requirements. 
The requirements can consist of needs, expectations and wishes. The next stage is the 
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transfer to an implementation-independent model of the future system, which descri- 
bes it and the degree of fulfilment of the requirements. In the last stage, the system 
functions would be identified and assigned to the elements of the system architecture, 
and interfaces and system boundaries would be defined. 

The other standards for software development that follow on from this will not be 
discussed in detail. They describe the relationship between RE, software development 
and formal vocabulary, but also methods for quality metrics or evaluation of tools for 
RE [15]. 

Methods for determining the knowledge of the stakeholders depend on the object 
of determination but also on the existing sources of requirements. Since this is a new 
development, document- and system-based methods will not be discussed further. Ne- 
vertheless, business process analysis as a document-based method should not remain 
unmentioned at this point. Suitable investigation techniques according to Rupp are, 
for example, questioning, observation or creativity techniques [16]. In the case of the 
former, surveys and interviews are the most important. A method for the next step of 
deriving requirements would be requirements analysis as a multi-stage process as well 
as testing techniques for the requirements, such as reviews, test cases, prototypes, and 
analysis models. In parallel, documentation and management should take place [16]. 


2.3 Documentation of Process Knowledge 


In order to record and document both the user and utility requirements resulting from 
the everyday scope of work, the aim was to be able to do this as intuitively and easily 
understandable as possible. The focus here was on the defined aspects of utility and 
usability. 

Since the requirements were recorded at the level of business processes and their 
activities, with a focus on the development of the model, this is the “design model” 
use case. According to Van der Aalst, this has a strong correlation with the choice of 
process modelling language [17]. The objective of modelling at the conceptual level 
is linked to essential criteria in the selection decision. These include ease of reading 
and interpretation, ease of learning and a wide range of application, so that after a 
comparison based on comparative studies, the decision was made in favor of BPMN 
2.0 [18]. 


3 Methodology 


The methodology presented below was developed in an exploratory approach using 
a focus group consisting of participants who record the needs of the users, process 
analysts and platform developers. The focus group participants were selected based 
on their insights and experiences that are highly relevant to the research. Their specific 
goals were to be incorporated into the method. As a research method, focus groups 
offer the opportunity to collect data on a specific concept, among other things, as is 
the case in our study [19, 20]. Due to the different points of view and perspectives of 
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the participants, communication and interaction in an organised discussion provides 
a flexible means of exploring new ideas or testing new concepts and thus ultimately 
generating data on the research question [19, 21]. The focus group sessions were con- 
ducted according to the procedure of Bär et al. in the context of various online appo- 
intments [22]. 

The focus group used in this study came from the KISS project, which aims to 
develop an approach for Al-supported digital platform services. The research project 
pursues the approach of enabling the rapid development of value creation networks 
by means of digital platforms in today’s times of dynamic change and thus impro- 
ving the ability of individual institutions to act. It aims to ultimately increase the re- 
silience of the manufacturing industry through the targeted AI-based networking and 
to improve the business capability of companies (in the project particularly in additive 
manufacturing with the goal of scaling to the manufacturing industry) as well as cus- 
tomer sectors affected by significant changes (in the project particularly in the medical 
and social sectors, possibly expanding to other sectors). However, before a compre- 
hensive technical and organisational concept can be developed for the implementa- 
tion and successful use of the Al-driven platform, it is first necessary to analyse the 
relevant ecosystem, considering the value network as a socio-technical system (orga- 
nisation, processes, data, IT). This results in a large number of relevant stakeholders 
for the platform, which must be integrated with their specific product, process and 
service descriptions to form the necessary value chains and analysed with AI support. 
The Supply Chain Operations Reference Model (SCOR) is a reference for the opera- 
tion, description and measurement of value creation networks and is therefore used 
as the basis for the integration mechanisms of the Al-based networking platform to 
be developed [23]. This is because the digital platform is supposed to generate these 
value chains and must therefore support the activities that, according to the SCOR 
model, most companies carry out in order to make their supply chains effective. Based 
on this, we argue that the process-based view of the different user groups is suitable 
to identify the services relevant for the digital platform and to provide them with the 
greatest benefit. Due to the prevailing diversification of the target groups relevant for 
the platform, a uniform process model must be created that ensures a common proce- 
dure for deriving platform-based digital services. 

In the following section, we therefore present the method developed in this con- 
text by means of a focus group study in a comprehensive way and also go into the 
individual steps of the procedure model. 


4 Approach for the Selection of Relevant Services 


Based on the discussions, a multi-stage process was developed in order to specifically 
focus on the user experience of future users in platform development (Fig. 1) 

The concept consists of five steps, whereby the first step is repeated iteratively in 
order to generate a comprehensive database. The individual steps of the concept are 
explained in more detail below. 
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Fig. 1. Concept for the selection of platform-based digital services 


4.1 Modelling the User Processes 


As already described, the method developed aims to derive meaningful services of 
a digital platform in a process-based manner. Accordingly, the existing processes of 
the potential user groups of the platform, which must be adequately recorded and 
modelled, form the basis of the process model. The aim here is to obtain as detailed 
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a picture as possible of the processes to be modelled in the institutions by means of 
process interviews. BPMN 2.0 has proven to be a suitable modelling language. These 
activity-oriented business process models (BPM) with the responsibilities structured 
in Swimlanes serve as a starting point to document the current effort or the possible 
need for utility in the respective institution. The process recording in the institutions 
is conducted in semi-structured interviews, which is a proven method in qualitative 
research. The questions are not rigidly prescribed, so that the dialogue between the 
interview partners is as free as possible, which is supported by an interview guideline 
with prepared questions structured according to topics (e.g., introduction, process ini- 
tiation, challenges in the process, conclusion). On the one hand, this interview guide- 
line serves as an orientation aid and for structuring the interviews. On the other hand, 
it also ensures the comparability of the interviews conducted. It is important that the 
questions are always formulated as openly as possible to stimulate free explanations of 
the processes and circumstances. Nevertheless, it is recommended to prepare so-called 
specifications, especially for the very specific questions about the process, which pro- 
vide approaches for targeted follow-up questions and accordingly help the interviewer 
to further specify the initial answers. While recording the process, individual and dou- 
ble interviews should preferably be conducted to prevent interviewees from withhol- 
ding information that they do not share with (larger groups of) third parties, such as 
superiors or colleagues. This goes hand in hand with the fact that persons with similar 
responsibilities and positions should always be interviewed across institutions to en- 
sure comparability of process views. Similarly, all interviews should be conducted by 
the same member of the project team to increase comparability between the interviews 
conducted. To be able to adequately create the individual BPMs afterwards, we recom- 
mend that the interviews are first recorded acoustically and then transcribed. 


4.2 Aggregation of Process Models 


In the next step, the process models created in the first process step are to be aggrega- 
ted into a generic BPM in order to combine the various individual processes and thus 
form activity clusters that are suitable for possible implementation as individual func- 
tion modules within a platform. By means of expert workshops consisting of process 
experts and experienced process analysts, a generic BPM can be derived that general- 
ises the individually recorded processes of the individual user groups to simplify the 
prevailing process diversity. At this point it should be pointed out that this aggregation 
step is of course accompanied by a certain loss of information from the individual 
process models of the individual institutions. Nevertheless, this aggregation requires 
the definition of framework conditions to be observed in order to systematically de- 
sign this process step. For this purpose, it is conceivable, among other things, to limit 
the number of activity clusters resulting from the aggregation of the process models 
in order to reduce the complexity of the aggregated BPM to a reasonable level. Furt- 
hermore, we recommend using the lanes as a criterion for the aggregation of activities 
and accordingly not merging activities across lane and thus role or department boun- 
daries, so that existing process interfaces remain intact. Taking such essential boun- 
dary conditions into account, two alternative procedures are available in principle in 
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the course of process aggregation. On the one hand, all captured process models as 
well as the activities they contain can flow equally into the generic BPM. On the other 
hand, it is also possible to prioritise the information that enters the process aggrega- 
tion, for example, to focus on more frequently occurring process paths or to trivialise 
infrequent activities. 


4.3 Attribution of the Complex Utility Function 


The previous steps describe an empirical actual state of activities and events of a syn- 
thesised existing process. In the following, we want to start from this modelled state 
and assume that an alternative “better” target process can be found, which is more 
useful on the one hand and is made possible using digital services on the other. Ma- 
nual processes should be simplified and substituted by digital sub-processes. 

The planned transformations should be cost-benefit efficient. For the improvement 
of total costs and total benefits, we want to make each activity, each event and each 
decision assessable in one step. We aim to formulate selective evaluation criteria to 
estimate the respective benefits at different levels. For example, we would assign at- 
tributes such as “temporal -”, “monetary -”, “crisis resilient —’, “emotional benefit” 
etc. to each activity. In addition to different benefit values of each element of BPM, 
we also estimate different cost values. Costs can be of various kinds. For target activi- 
ties to be evaluated, implementation, establishment and operating costs are incurred, 
among others. 

Costs and benefit values do not necessarily have to stand in relation to each other 
individually. In the next steps, they are used to evaluate potential transformations in 
terms of whether the overall cost-benefit ratio would increase. To determine this, we 
propose simulations using a complex benefit function that integrates all cost and be- 
nefit attributes in a weighted way. In the simulation, BPM instances are processed, for 
which additional measured or estimated knowledge about flow distributions is needed. 

From a scientific point of view, we find it interesting to investigate the methodo- 
logy of determining suitable benefit attributes and the joint calculation in a complex 
benefit function regarding the user experience. The result is an extended and assessa- 
ble BPM. 


4.4 Classification of Possible Function Modules 


In the next step, various alternative processes and sub-processes are to be developed 
and discussed with the help of the previously developed evaluation methods. 

New substituted activities proposed by experts need to be estimated in terms of 
their costs and benefits in the same way as existing activities. It should be noted that 
the transformation into an improved process is associated with one-off costs. These 
costs include both the one-off potential development costs of the digital services and 
the overcoming costs that all the entities involved must incur to accept the change 
and establish it in the long term. As an example, a potential actor of a platform has to 
actively decide to give up his usual processes and get motivated to learn new procedu- 
res, etc. These costs should be considered in the analysis. If necessary, it may even be 
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necessary to model intermediate transformations, because it is only through these that 
the new target process appears achievable. All in all, a set of new imaginable target 
processes should be modelled, which are more useful, more accepted, and not more 
cost-intensive over a defined period. 

Our focus is on categorising all variations of individual functionally definable 
sub-processes individually and in combination. The resulting set of different potential 
overall processes can then be ranked using the proposed evaluation. We finally select 
the best-rated, most useful, and feasible alternative process including the new sub- 
processes. 


4.5 Development of the Requirements for the Target Process 


The last step in the process model describes the derivation of the requirements of the 
target process in the context of a focus group. According to Rupp, this can also be 
seen as a review of the requirements [16]. This workshop takes place with both deve- 
lopers and stakeholders — the future users. The users should have the same role and 
sector affiliation as in the process interviews, since they were identified as relevant 
stakeholders at the beginning. In the focus groups, it will be examined whether the 
classified process works with the future role distributions on the one hand, and on the 
other hand, which building blocks could be considered for a possible implementation. 
This functional selection of possible services could then be the starting point for fur- 
ther development and keep the effort low for all those involved. 


5 Conclusion and Outlook 


In general, digital platforms and the ecosystems that are inevitably linked to them are 
not only part of a wide variety of research disciplines, but they have also aroused great 
interest in the past, especially in business practice. Nevertheless, the systematic de- 
rivation of beneficial services for digital platforms, especially in connection with the 
use of AI, remains undeveloped. We therefore propose the idea of a process-based ap- 
proach to deriving platform-based digital services, which is reported in this paper. In 
this context, the findings from this focus group study are not only of interest to design 
scholars (the authors or others) who are interested in developing such a platform and 
thus in designing the method. They equally contribute to pointing out a development 
and implementation agenda for future users of such Al-based platforms and their de- 
velopers. Focusing on utility and usability with an emphasis on subjective utility and 
efficient goal attainment can effectively narrow the solution space to be considered as 
an aid in this regard. 

However, the process model just presented not only provides the basis for deriving 
suitable digital services for an Al-based platform on the basis of real processes. On 
the other hand, it also itself presents various potentials for the use of AI to support the 
individual steps of the method, which are currently envisaged within the framework 
of empirical methods and would consequently experience a significant reduction in 
workload through applications of AI. For example, the second step of the procedure, 
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in which the aggregation of the individual process models (see 4.2) of the instituti- 
ons under consideration takes place, opens up the possibility of carrying this out in- 
dependently, e.g. rule-based, analogous to the SCOR model. Likewise, the fourth step 
in the classification of possible function modules (see 4.4) shows another possible use 
of Al solutions in the context of a cluster analysis. This already shows the first start- 
ing points for the further development of the method presented. Although the process 
model is undergoing initial validation as part of the KISS project, the design process 
has not been fully completed and thus creates the starting point for further research. 

Our research focus in this work is on the elaboration of digital processes as part 
of the user experience rather than the design of the user interfaces. It should also be 
mentioned that the respective framework conditions are a considerable limiting factor. 
On the one hand, only a predefined section is surveyed during the process recording 
in interviews and thus all further results are pre-determined to a certain extent, and 
on the other hand, the number of decision-making options within each step is limited 
by the framework conditions, so that it may well be that no functional changes are 
considered beneficial. The composition of the respective focus and expert groups also 
determines success. 
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Check for 
updates 


Classification of Static Poses Based on 
Key Point Detection for Application of 
Incriminated Image Files 


Schönbrodt Antonia? ®)® 


1 DigiFors GmbH, Torgauerstraße 231, 04347 Leipzig, Germany 
? Springer Heidelberg, Tiergartenstr. 17, 69121 Heidelberg, Germany 
info@digifors.de 
https://digifors.de/kontakt/, 
http: //www.springer.com/gp/computer-science/lncs 


Abstract. The role of artificial intelligence, particularly in enhancing 
decision-making processes and facilitating automation, has become indis- 
pensable in today’s society. Law enforcement agencies have used the find- 
ings of this research for several years to analyze the ever-growing volumes 
of data. This approach owns an important role in the detection of child 
and adolescent pornography. Research conducted in this field has tradi- 
tionally been dependent on the findings from skin filtering studies. How- 
ever, some scientific publications and empirical studies reveal insufficient 
classification when using nudity levels. This work takes a more promising 
approach by leveraging the knowledge of an existing system and combin- 
ing it with a homegrown model to improve the detection of incriminated 
image files. For this implementation, techniques from motion analysis 
were used to open up a new field of pose recognition. 


Zusammenfassung. Die Rolle der künstlichen Intelligenz in Bezug auf 
ihre Individualität und Automatisierung ist in der heutigen Gesellschaft 
unverzichtbar geworden. Die Ergebnisse dieser Forschung werden seit 
einigen Jahren ebenfalls genutzt, um Strafverfolgungsbehörden bei der 
Analyse der ständig wachsenden Datenmengen zu unterstützen. Dieser 
Ansatz besitzt eine wichtige Rolle bei der Aufdeckung von Kinder- und 
Jugendpornografie. Bisherige Studien in diesem Bereich stützen sich 
größtenteils auf die Ergebnisse von Studien zur Hautfilterung. Einige wis- 
senschaftliche Veröffentlichungen und empirische Studien zeigen jedoch, 
dass die Verwendung des Nacktheitsgrades keine ausreichende Klassi- 
fizierung ermög-licht. Diese Arbeit verfolgt einen vielversprechenderen 
Ansatz, indem sie das Wissen eines bestehenden Systems nutzt und es 
mit einem selbst entwickelten Modell kombiniert, um die Erkennung 
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inkriminierter Bilddateien zu verbessern. Für diese Implementierung 
wurden Techniken aus der Bewegungsanalyse verwendet, um ein neues 
Feld der Posenerkennung zu erschließen. 


Schlüsselwörter Artificial intelligence - image categorization - pose 
detection - data pre-processing 


1 Introduction 


The idea of using artificial intelligence to assist humans and introduce new inno- 
vations has been around since its inception in 1950 [1]. Its use has already consol- 
idated and is constantly expanding in many fields such as business, the automo- 
tive industry, and many others [2]. Nevertheless, the field of artificial intelligence 
is not sufficiently considered in the context of law enforcement. As digitization 
continues, ever-increasing amounts of data are being generated that cannot be 
handled by law enforcement’s own resources alone [3]. In order to facilitate the 
analysis process for investigators, a field opened up in academia a few years ago 
in which neural decision patterns are used to examine and evaluate the volumes 
of data. The surge in data volume is especially noteworthy in less-visible crime 
areas. This is also true for the analysis and detection of child and adolescent 
pornography [4]. Here, not only can the analytical aspects be made more effi- 
cient through the use of neural networks, but artificial intelligence is also used 
to minimize the psychological burden on investigators. However, most of the 
models to be tested are currently still in the testing phase. For this reason, a 
bachelor’s thesis was written to provide a foundation for the research listed here. 
It is intended to change the perspective regarding the use of artificial intelligence 
for the detection of child and adolescent pornography. To this end, existing meth- 
ods have been partially redesigned. This approach makes them more resilient to 
large amounts of data, as this adapts the methods to other subjects. In this way, 
the constructed systems can be adapted to the different needs and interests of 
the users in order to revolutionize the analysis of the addressed offense domain. 
Thus, the goal of this thesis is to address the question of whether criminal image 
recognition can be improved by allowing knowledge of pose recognition to have 
a decisive influence on the determination of incriminated images. The basis for 
this theory is the aforementioned bachelor thesis, which was further adapted and 
improved after its completion. 


2 Scientific Framework 


2.1 State of Research — Artificial Neural Networks for Child 
Pornography Detection 


The emerging field of artificial intelligence in science now makes it possible to 
analyze large amounts of data faster and more efficiently and in a more tar- 
geted manner[5]. The most promising scientific contributions are currently still 
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in the research stage [6]. In preparation for the aforementioned bachelor’s thesis, 
systems dealing with this area of crime were examined in a practical manner in 
police agencies. None of the models tested were targeted, as most research in this 
area relied on the use of the skin filter alone. A suitable scientific publication 
in this regard was written by Nicole Garbers and Michael Brodthage[6]. This 
paper delves into the challenges posed by employing artificial intelligence in the 
detection of child pornography content. In their 2021 execution, they describe 
the limitations of the skin filter often used for classification purposes and their 
improvements. According to them, the skin filter by itself does not provide suit- 
able preselection [6]. One problem that arises from the nudity level classification 
is that the neural networks are not able to recognize different ethnicities [7]. On 
the other hand, misinterpretations of the analyzed data also occur in everyday 
situations of the data under analysis. This phenomenon of misclassification also 
occurs when selfies are taken. Since these image recordings represent a high per- 
centage of skin of the entire image [8]. In addition, borderline cases often occur 
on the evidence being analyzed, where underage individuals are required to pose 
sexually clothed. These considerations show that the skin filter determinations 
for pornography detection [9] cannot be used alone in such cases. Moreover, 
the described views of Garbers & Brodthage could be confirmed by the results 
of the apparent observations of the practical models. The multilayeredness and 
complexity of the phenomenon area to be addressed and the listed limitations 
of Gabers and Brothage have the consequence that the analysis of the nudity 
level has to be improved by considering further characteristics. Since final thesis 
only provide a limited framework for research, the considerations in this area 
must be narrowed down. The scientifically most important area is provided by 
the motion analysis with the analysis of the joint points [10], [11], [12]. Despite 
the scientific dissemination of this topic, the determination of poses and their 
classification was not recognized in court for a long time. This changed with the 
49th Criminal Law Amendment Act [13]. Through this amendment, recordings 
without a direct sexual act are now considered criminal. It is now sufficient only 
to detect the pose itself [13]. Based on the arguments described above, the ques- 
tion of whether the detection of criminally relevant pornographic material can 
be improved by determining and classifying static poses will be explored. 


2.2 Pose Detection 


According to the Criminal Code [§184 para. 1 no. 1 lit. B], posing of a partially 
or fully clothed child is punishable. In this context, the legislator defines posing 
as an “unnatural and sexually accentuated posture”. In this category, children 
are induced to assume sex-related and stimulating postures in order to imitate 
adult models [13]. For the detection of posing, a uniform determination scheme 
must be established first. In the context of this work, unnatural poses are static 
postures that do not correspond to the typical behavioral patterns of children. 
This category includes positioning the buttocks in front of the camera, rolling 
the body on the floor and spreading the legs. In addition, there are targeted 
postures in which certain body regions are deliberately accentuated by certain 
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inclinations of the arms or legs. Furthermore, images in which touching the chest 
or intimate area is in the foreground are counted under a gender-emphasized pos- 
ture. According to [14], tracking and estimating the human pose is a computer 
vision task. The basis of this is the recognition, assignment and tracking of key 
semantic points. There are already exist various software applications to sup- 
port the fixing of joint points and thus the determination of poses. During the 
research period, different applications were considered and evaluated. Various 
pose detection systems were tested for the planned procedure and were adapted 
to the conditions. The major known pose detection systems AlphaPose [15] and 
Mask R-CNN [16] are not suitable for the intended classification. Mask R-CNN 
is based on semantic segmentation, which can lead to shallow human bounding 
boxes and this leads to a incorrect classification of joints. Furthermore the system 
was fundamentally not designed for pose estimation [17]. The AlphaPose pose 
estimation system is rather designed for the determination of multiple individ- 
uals. The scientific review by Siddharth Sharma also mentions that this model 
provides more failure cases than other models in pose estimation [17]. In order 
to actively compare the favored pose estimation system OpenPose with another 
suitable system, local tests between OpenPose and another system called Medi- 
aPipe were performed as part of the bachelor thesis developed. It has been shown 
that OpenPose can detect the skeletons more reliably. In particular, hidden body 
parts could be detected better as with MediaPipe due to the additional function 
of occlusion. 


2.3 OpenPose 


OpenPose is a pose recognition system [18] which was developed based on the 
COCO dataset. OpenPose is the first “open source real-time system for 2D pose 
estimation” [18], which is able to determine key points of the human body. 
The keypoints describe various points that are placed at significant locations 
on the body, such as the shoulders, knees, and elbows. [19] The basic principle 
of the convolutional neuron network based approach is the so-called bottom-up 
approach. Here, in a given image, the different keypoints are first determined 
and then the differently recognized skeletal parts are combined to form a whole 
object [19]. Such approaches exhibit a certain robustness and have the potential 
to decouple the influence of runtime complexity from the number of people [20]. 
OpenPose provides the first bottom-up approach in which association values 
are used to encode an image in terms of limb position and orientation via so- 
called Part Affinity Fields (Pafs) [18]. Here, the Pafs are defined as a set of 2D 
vector fields consisting of a series of flow fields. [18]. After the initial estimation 
of these vectors, the information is improved based on the multilevel classifier 
at each refinement level. Subsequently, a grouping procedure is performed in 
which the vectors corresponding to the same joint region are connected and 
ranked according to affinity. This principle is based on the idea that the joints 
to be detected are connected by limbs. Finally, the detected points are connected 
to represent the pose key points. First, the connections of one group of joints 
are performed and then those of the other limbs are determined [18] Thus, the 
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connection of the body-and-foot keypoints detector finally provides a complete 
human skeleton with at least 25 different keypoints, which can be used for further 
considerations. 


3 Methods — General Procedure 


3.1 Assembling the Data Material 


Different data sets were compiled for pose recognition and neural network train- 
ing purposes. Given legal and ethical constraints, we cannot use incriminated 
image files for creating suitable training material, as the possession of child and 
youth pornographic content remains prohibited, even for scientific endeavors [12]. 
For the work with artificial neural networks, various data sets are needed, which 
due to their diversity ensure a robustness of the network against large amounts 
of data. Initially, we require a dataset tasked with training and updating the 
neural network. In order to achieve diversification of the dataset, care must be 
taken to use shots of people of different ethnicities, group shots, and backgrounds 
in the classes under consideration [6]. In addition, diverse shooting angles and 
situations have to be considered in the image scenes to under consideration. The 
freely available collection of Figshare [21] was used as the pornographic data set. 
However, due to this, the number of usable image files reduced rappide. Since 
the goal of this research is to distinguish between posing and no posing images, 
everyday images were added to the dataset as an opposing class. Thus, both 
domains form the training dataset used in this work. Considering the limited 
time for the underlying bachelor thesis and the limited datasets, only the small 
amount of data was relied on. In order to check how well the learned model gen- 
eralizes the data or if it represents only the training data, a validation dataset 
is necessary. For this purpose, the corresponding data from Figshare [21] was 
also taken into consideration. Then, another independent dataset is needed to 
test the neural network. The files from the test dataset of Figshare [21] are not 
only pornographic images, but also everyday clippings. The use of both classes 
is necessary to test the functionality of the artificial intelligence related to the 
objective. Only in the comparison between non-pornographic and pornographic 
poses it can be seen if the system performs a good classification. 


3.2 Created Workflow 


The scientific thesis of the underlying bachelor thesis is to be able to make 
a classification of sexual poses on the basis of defined cornerstones. For this 
purpose, a workflow adapted to this topic was developed, which already uses 
an existing Al-generated innovation and combines the basic knowledge with a 
self-created instance. 
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Fig. 1. Visualization of the created pipeline with included integration of the two neural 
systems 


This compiled construct provides information about whether a human pose 
is understood as posing or not posing. The listed parts were combined in a 
Python code to form a packaged analysis system. The architecture of the work- 
flow is shown in Fig. 1. At the beginning, the pipeline is briefly summarized 
for a better understanding. First, the image files to be analyzed are passed to 
the program. In this process, the files are first examined using the embedded 
framework OpenPose, where the person-based rigs are created. The results are 
optimized with respect to the objective. Subsequently, they are passed to a 
neural network, which was constructed self-referentially for the problem to be 
analyzed. The structure of this system is described in more detail below. This 
network divides the image data into the classes posing and no posing and then 
assigns them to equal folders, which can be viewed at the end of the pipeline 
with the respective results. The entire process was visually structured so that 
at the end it is comprehensible which image files were classified as posing and 
which need to be examined more closely by the officials. The output of the work- 
flow is therefore not the human skeletons created in the intermediate steps, but 
the data originally entered. This form of representation ensures that each image 
can be uniquely assigned. Thus, the decision of the neural network can be bet- 
ter observed and evaluated. Possible misinterpretations in the representation of 
the original files can be better detected. Thereupon, a faster identification and 
adjustment of the inaccuracies can take place. These deviating results are then 
used to be integrated into the new learning process of the neural network and 
to improve it. 


Prediction of person-specific body skeletons by OpenPose. The Open- 
Pose motion analysis system was used for pose determination. In this study, 
only the demo version of this implementation was used, as it does not rely on an 
NVIDA graphics card, unlike the more comprehensive version. The initial testing 
did not yet yield the possibility to test the encryption results with the full ver- 
sion. Whether changing the way the program is run improves the results of this 
work will be tested in further future research. Through the demo version used, 
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human skeletons, also called rigs, are generated from the incriminated data. To 
better narrow the predefined skeletons for the range of phenomena to be consid- 
ered, slight changes were made to the implementation code line of OpenPose. 


Listing 6.1. Representation of the customized demo code line from OpenPose 


1 bin\OpenPoseDemo.exe —image_dir examples/media/ 
2 ——disable_blending — hand ——write_image folder_path 


The parameters were changed by the attribute disable_blending to remove the 
background from the images to be analyzed. This change allows the neural net- 
work to analyze only the keypoints and avoids distortions caused by accentu- 
ated backgrounds. In addition, the number of keypoints to be classified has been 
increased by also inducing the detection of the feet and hands. The foot detec- 
tion is already integrated by default in the demo version. With the attribute 
-hand the hand detection was added to the determination. These body regions 
play an important role in the detection of posing images. Since especially hands 
can provide clues for a conscious accentuated movement. However, not all poses 
are uniquely recognized when OpenPose detects the key points. For example, 
there are often deviations when the spine is bent, the legs are stretched in the 
air, or the buttocks are held up to the camera. Currently, attempts are being 
made to counteract this problem by specifically training the neural network with 
such borderline cases in order to promote the robustness of the network. On the 
other hand, efforts are also underway to develop the Linux-based variant of the 
workflow to incorporate the deeper innovation of OpenPose. 


Labeling of the inserted data sets — preparation of the training phase. 
To initiate the determination of the poses, they must first be precisely evaluated 
and categorized for training purposes. These differences are then learned by 
the neural network so that it can determine a suitable classification for unknown 
image objects. Reliable preparations are pivotal for the success of training. After 
the initial billfiles were pipelined, OpenPose added keypoints to them, and due to 
the change in code implementation, the background was subsequently removed. 
All that is left is the pure skeleton. Subsequently, in preparation for the training, 
the values of individual skeletons must be established. For the labeling of the 
training data exclusively results are used, with which OpenPose could detect 
nearly all key points. Such well-detected skeletons include outputs in which there 
are hardly any deviations from the original keyponits detection. The absence of 
single small but still visibly complementary body parts, such as the absence of 
a lower leg, is within the tolerance range. A visualization of such good results is 
shown in Fig. 2. Based on this selection a robustness of the later trained network 
is generated and misinterpretations are prevented. 


72 S. Antonia 


„badly detected“ 


/ Fa LabellMG 
——, partially detected“ PFN r 


rm 


train the network 


„completely detected“ 


7 | labelling the best results for class determination 


selection of best results 


Fig. 2. Visualization of labeling approach in preparation for neural network training 


For the assignment of the two classes posing and no posing the method of 
labeling is used. In this method, characteristic sexual poses are selected and 
assigned to the class posing. For the classification of data, different applications 
exist to determine the respective images or to assign them to different classes. 
For this work the tool LabelIMG was used. With this tool the image files with the 
most recognized keypoints are read in and assigned to the desired class by hand. 
The file is provided with a bounding box, similar to a bounding box. This frame 
is then assigned the class posing or no posing. This assignment works similar 
to the assignment of object views, where objects such as ships are assigned to 
the class ship due to their nature. The class information and coordinates of the 
boxes are saved and stored in an xml file. These for the further procedure used 
and afterwards the own developed net passed. 


Construction and training of the neural network As a proof of concept 
for classification, a simple neural network with a convolutional layer based on 
the existing OpenPose system was attached to process the results of the herding 
system. The results of this network, however, did not provide satisfactory results, 
because due to the architecture and the resulting output only one class was 
detected. As a result, the proof of concept was adapted, revised and transformed 
into a convolutional neural network with pooling layers during the course of the 
research. The individual layers of the self-developed transformed network are 
shown in Fig. 3. To increase the processing power of the system, the output 
image files of the previous mesh were initially scaled to an input value for the 
new mesh of 64 x 64 pixels using the Resize (64,64) function. This resolution 
was chosen to increase the processing power of the mesh. 
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Fig. 3. Representation of the individual layers of the specially created neural network 


Previous studies frequently utilized higher resolutions as they facilitate the 
extraction of more information from an image. Nonetheless, as this study is pri- 
marily focused on pose detection rather than detailed image analysis, a lower 
resolution was selected. In addition, the input layer is represented by 3 layers, 
for the image colors red, green and blue. These pixels are responsible for mak- 
ing colors visible on the input files. After the input layer, two Convolutional 
Layers are added. Their output is determined by the so-called ReLu function. 
They follow the principle of feature learning. Subsequently, the data is reduced 
by the adjacent pooling layer. This is done by filtering out the strong features 
and discarding the weak features. At the end of the network there are two linear 
layers, which are transformed into a one-dimensional layer with the help of the 
view() function. These layers are finally responsible for the classification. After 
defining the neural network structure, the forward method is applied in the pro- 
gram code to connect the network architecture. It also enables the execution of 
the forward pass. Consequently, the layers process together until the output is 
available, which provides the image. In the output layer, the softmax function is 
used as the activation function instead of the previously used ReLU function. In 
this research, the softmax function was deliberately chosen because it is specif- 
ically used for classification problems. Many multilayer neural networks end up 
with real-valued results output at the last hidden layer. These often pose scaling 
challenges for the neural network, creating obstacles for further processing. Here 
the softmax function is helpful, since it converts the results into a probability 
distribution, which can be better used by the network. For the analysis process, 
the zero points of the gradient and the activation of the weights are determined. 
Using Pytorch’s Autograd module, differentiations are performed and weights of 
the model are automatically updated. In order for Pytorch to not only determine 
whether the network’s predictions are true or false, but also to determine their 
valence, a loss function is needed. For this study, we utilized the CrossEntropy 
function. This is also used in other work for classification tasks. The loss func- 
tion is used during the training loop to match the predictions with the actual 
labels. This information is backwarded by the backward() method to allow the 
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initial layers to learn from the differences and update the network. Through the 
optimizer() function, the differences are passed to the gradient, resulting in an 
adjustment of the weights at each layer. For the training loop, the parameters 
just described are passed to the network, which is shown in the underlying rep- 
resentation. In the code sequences, Cnnet was used to describe the self-created 
neural network. 


Listing 6.2. Introduction of loss function and optimizer into training 


1 optimizer = optim.Adam(cnnnet.parameters(), lr=INIT_LR) 
2 loss_fn = nn. CrossEntropyLoss () 


Due to the given resources and the limited time for this research, an epoch of 50 
was decided upon. This implies that the training process iterates 50 times. With 
this number, representative results are already produced, which, however, do not 
yet drastically extend the analysis process. After all, the accuracy of the training 
as well as a loss function were determined in order to identify and subsequently 
evaluate the functionality. In summary, the training can be described as the 
following lines of code. 


Listing 6.3. Representation of the training loop 


” 


3 for data, target in tqdm(train_data_loader , desc= 
Training...”): 

4 optimizer. zero_grad() 

5 output = cnnnet (data) 

6 loss = loss_fn (output.squeeze(), target. float ()) 

7 loss . backward () 

8 optimizer.step() 

9 #calculate the training loss 

10 H[” train_loss”].append(loss.item()) 

11 train_correct += (output.argmax(1) == target .argmax(1)). 
type(torch. float ).sum() .item() 

12 #calculate the accuracy after training loop 

13 _train_correct = train_correct / len(train_data_loader. 
dataset ) 

14 H[” train_acc”].append(_train_correct » 100) 


However, if only training data is used for programming, the neural network used 
may overfit. This means that the system is good at recognizing and evaluating 
representations that it has been trained with. However, generalized represen- 
tations that are not included in the training dataset cannot be analyzed. For 
this reason, other datasets such as the validation dataset and the test dataset 
are mandatory. To fulfill this scenario, a validation loop is created after the 
training loop. This can ensure that the model is not over-fitted to the training 
data and recognizes not only them, but also independent data. The training 
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procedure just described is repeated with the validation data. Once again, the 
dataset undergoes batch-by-batch processing within the system, with the loss 
being calculated. However, no backward() method and optimization approaches 
are needed here, since the parameters of the model do not change in this step. 
A similar procedure is also used to pass the data to the test loop. Evaluation 
is performed independently of training or validation. In order to determine the 
learning and recognition success of the respective phases, the accuracy of the 
respective predictions were measured for the training and validation data set 
and were output after the respective phases. Thus, after each training phase, 
a percentage is shown which indicates how well the system classified the data. 
Afterwards, the trained model can be passed to the pipeline. 


Composition of the individual components After training the self- 
generated neural network, the two systems must be connected to form a single 
unit in order to pass the results of the motion analysis system to the classifier. For 
this purpose, the two sequences must be merged in the code pipeline. In order to 
be able to guarantee further modifications of the created network, the architec- 
ture and the training of the network were deliberately combined into a separate 
program code, which was subsequently saved in a file named new_model.pt. This 
project file is then passed to the pipeline in only one program line. This ensures 
that other possibly better meshes can be trained separately and can later be inte- 
grated into the pipeline without much effort. Thus, the pipeline can be quickly 
and efficiently replaced by even better neural networks, increasing its individu- 
ality. Based on the training, the neural network can now evaluate the captured 
image files and assign them to the two classes Posing and no Posing to be deter- 
mined. This assignment is done by sorting the images of the respective classes 
into two folders of the same name. Since the output of the neural network is the 
individual skeletons without background, these files should actually be in the 
folders of the same name. According to the workflow, these folders now contain 
the images in the form of skeletons. However, if you are not intensively involved 
with the subject, you will not be able to do anything with this kind of repre- 
sentation. In the meantime, a program has been created that ensures that the 
converted files are back in the folders as original files at the end of viewing. How- 
ever, OpenPose automatically changes the file extension of the images as soon as 
the skeleton is determined. Thus, standard extensions like .jpg, .png or .bmp are 
replaced by _rendered.png. This procedure makes it difficult to reconstruct the 
original files. In the underlying bachelor thesis, an attempt was made to solve 
the problem by removing the file extension _rendered.png and replacing it with 
some kind of tariff system with .png and .jpg. This led to the fact that 34% of the 
image files to be analyzed could not be determined. This is because these were 
not taken into account when restoring the image extensions. To counteract this 
problem, in later studies a working directory was created containing a folder in 
which all image files were stored again before the corner points were determined. 
In this process, all image files were given a duplicate file extension. This means 
that the file neuronalesNetz.webp became neuronalesNetz.webp.webp after this 
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step. This newly named file is now packed into an intermediate folder “Zwischen- 
stand”. Afterwards, the data is passed to the OpenPose system for item recog- 
nition. The file extension is now replaced by the predefined phrase again. Thus, 
the file name is now neuralNet.webp _rendered.png. With this in mind, a Python 
program was written with the Pose Estimation System extension removed. What 
remains is the extension of the original file. So the ending is known and at the 
end all files can be sorted into the respective label folders again. 


4 Results 


The results of the neural network training provided satisfactory results. At the 
beginning of the training, the accuracy of the predictions was only 50.75% for 
the training data set and 52.64% for the validation data set. These values could 
be improved strongly after the 50 training runs. It was found that the model 
predicted the values of the training data set with an accuracy of 97.59%. The 
accuracy for the fit to the validation data set was 98.51%. The loss functions 
also obtained improved from 0.687834 to 0.085758 for the training data set 
and from 0.702857 to 0.024533 for the validation data set. The total train- 
ing time was 3425.92 seconds. By restructuring and adjusting the pipeline, the 
developed workflow could be improved. For testing purposes, the corresponding 
reduced dataset from Figshare [21] with 532 data was introduced into the created 
pipeline. 


Current Research Results 


Number of detected files 
N 
8 


posing no posing 
Gcorrectly classified 229 134 
m misclassified 153 16 


Fig. 4. Graphical representation obtained of the classification results 


Due to the change to determine the file extensions, it was now possible to 
recognize and subsequently determine all image files from the constructed model, 
unlike in the underlying bachelor thesis. In total, 150 images were placed in the no 
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posing folder and 382 images were placed in the posing section. Fig. 4 illustrates 
the exact results. It was found that the basic idea of this research, to detect 
posing on the basis of pose estimation systems, could be applied to many images. 
In addition, the consideration of the revised output representation could be fully 
implemented. It was shown that the developed pipeline could reliably classify 
posing images as such even if the depicted persons showed postures where, for 
example, the legs are spread or the buttocks are held into the camera. However, 
the developed system faced challenges in recognizing postures involving bent 
legs and a curved spine This finding is reflected in the 16 misclassified image 
files in the no posing category. The image files there predominantly illustrate 
the problem just described. In order to minimize this already known problem, 
the neural network is currently being trained with such problematic image files 
in order to improve the recognition rate of these poses. On the other hand, it 
has been shown that the trained neural network exhibits a slight over-fitting. 
This means that many apparently everyday poses, 153 in total, were sorted into 
the category posing. In most of the images, the people are working with their 
hands to express certain situations. Due to the targeted training on the hand 
region to detect a playful accentuation of the hands, certain hand positions were 
misinterpreted by the system. In addition, the studies conducted showed that 
the OpenPose system, which was integrated first, had problems identifying the 
corresponding skeletons for persons without a recognizable eye region or with the 
half of the body covered. For example, no skeletons existed for individuals with 
no eyes visible. As a consequence, the subsequently attached neural network fails 
to detect potentially sexually suggestive poses as posing, if Open Pose doesn’t 
provide a substantial skeleton beforehand After the results were obtained for 
applied incriminated dataset with adult subjects, a dataset with everyday image 
files of children was injected into the pipeline. This step was performed in order to 
verify whether the transfer, of the already confirmed functioning of the pipeline, 
is also transferable to the proportions of children. Particular attention was paid 
to the fact that the interpretation of skeletal parts was transferred. The tests 
showed that both systems implement the learned factors and adapt them to the 
respective image files. 


5 Discussion and Limitations 


The results of the research have shown that motion analysis methods can also be 
used for the detection of sexually suggestive poses. Furthermore, the generated 
pipeline proved that the estimation system just described can be combined with 
another neural network. Since the present work deals exclusively with the anal- 
ysis of movements with a pornographic context, a fundamental improvement in 
the recognition of child and adolescent pornographic files cannot be confirmed 
by this one method alone. However, to further fill the gap in scholarship in this 
area, the proposed methodology must be combined with the other preliminary 
considerations from the research. This work will pave the way to bring more 
attention to the issue of child and adolescent pornography and provide further 
suggestions for the outdated rating systems. 


78 S. Antonia 


5.1 Classification Problem 


An existing problem, which has been shown in the course of this research and on 
the basis of the results, is that the self-created model can only act in combina- 
tion with previously switched network OpenPose. This means that if OpenPose 
provides weak or no keypoints, the subsequent model cannot perform reliable 
classification either. Although the supposed best evaluation system was cho- 
sen for this research, it also has its limitations. The system was developed to 
recognize human poses and make predictions. However, there are no scientific 
publications yet on whether this system is also used to detect sexual posing. 
Thus, the present results are the first attempt to implement the prediction of 
skeletons by OpenPose in a different domain. It could be confirmed that Open- 
Pose shows problems when the spine of the respective person is curved. Thus, 
the results were only inconsistent for certain lying positions. In addition, images 
where no eyes are visible cannot be skeletonized. This means that close-ups of 
buttocks or genitals are not analyzed when posing. These inconsistencies might 
have arisen because the actual functionality of the rating system is to recog- 
nize people in everyday situations. Should OpenPose become further established 
for the analysis of pornographic files, it would be possible to revise individual 
program lines to adapt the recognition to the problem being analyzed. In this 
context, it would be worthwhile to investigate in future research how these dis- 
crepancies can be reduced, or even whether there are other systems that better 
perform the task of evaluating sexually suggestive poses. This issue could also 
be mitigated by enhancing the analytical capabilities of OpenPose. On the one 
hand, this can be done by incorporating the deeper version of this initialization 
into the pipeline rather than the demo version. The demo version seems to have 
problems recognizing certain postures. The full version method has not yet been 
implemented because the computer used to obtain the data does not have an 
NVIDA graphics card. For this reason, the demo version was first integrated 
into the pipeline, as no extensive technical requirements had to be met for this. 
On the other hand, the network must be specifically trained with such images, 
which were incorrectly recognized by the results or where the skeletons were 
only partially displayed. In this way, an attempt is made to improve the results 
in further research. The mentioned over-fitting of the neural network was char- 
acterized by an overtraining of the hand position. The results and the output 
image files have testified that many files were sorted into the posing category on 
basis of accentuated hand positions. The over-fitting should be reduced by using 
a much larger amount of data for further training and by setting the poses to be 
labeled more expressively. This adaptation should prevent the artificial intelli- 
gence from deciding whether a pose is sexually stimulating or not based on arm 
positions alone, but always acting in combination with the lower body region. 
The neural network was almost one hundred percent accurate in categorizing 
poses with spread legs into the posing category. 
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5.2 Work Restriction 


The aforementioned classification problem can also be mentioned as a limitation 
of this work. Another limitation of this work is that a lack of available data sets 
hampered the analysis of the research objective. There was a lack of reliable and 
problem-specific material. Although the datasets used, each containing approxi- 
mately 3,000 files, already contained less data than comparable datasets used to 
train other neural networks, the datasets used had to be further reduced. Deep- 
ening the dataset was not possible due to limited time, as the image files to be 
classified had to be labeled individually by hand. With the reduction of the data 
set, it had to be accepted that the neural network would not achieve the desired 
results. If the network would be trained with more data, a higher variability 
would arise and thus a higher probability would be given to recognize other 
poses. Furthermore, due to the legal situation, the main task, an improvement 
for the detection of child and youth pornography, could not be fully met. The 
developed pipeline has shown that sexually suggestive poses are classified into 
two categories posing and no posing with regard to the created workflow. How- 
ever, since a one hundred percent transfer of the workflow to child pornographic 
files could not take place, only the recognition and marking of child skeletons was 
tested. This ran without problems. Based on this assumption, it can be assumed 
that the assumptions made here can also be transferred to the topic of posing in 
relation to children and adolescents, since everyday child skeletons were already 
recognized by the system. 


6 Summary 


The present study aimed to design awn worklow to improve the detection of 
child and adolescent pornographic files. To this end, insights from the field of 
motion analysis were to be used to distinguish posing images from everyday 
images based on classification. To answer the guiding question, a motion analysis 
system was connected to a neural network to draw conclusions about the valence 
of the data. It was found that OpenPose was the best motion analysis system of 
the systems tested to detect sexually suggestive poses. The meaningful results 
confirmed that the combination of the two models into a pipeline is suitable 
for detecting sexually suggestive poses of adults. In addition, the self-created 
neural network was integrated into the pipeline in such a way that it could 
be easily replaced based on the modification of a single program line. Thus, 
better and deeper neural networks could be inserted into the already working 
pipeline without a big effort. Thus, better and more accurate results could be 
achieved. Tests have also shown that the system can also be applied to the 
physique of children. For this reason, it can be assumed that the findings for 
sexually suggestive poses can also be transferred to the physique of children 
and adolescents. Thus, the basic goal of this thesis could be achieved and the 
research question could be confirmed. Furthermore, the recognition of posing can 
be used in other areas as well. For example, the learned knowledge from this work 
could be used to recognize intimacies between people, since elements of posing 
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appear in many actions between individuals. For example, spreading the legs and 
positioning the buttocks in front of the camera occur more frequently in such 
scenarios. To subsequently implement this idea with the present pipeline, a more 
elaborate network would need to be created and trained with appropriate data. 
Afterwards, the model is saved and passed to the created pipeline in the form of 
a program line. In addition, it should be noted that the findings of this research 
alone cannot improve the analysis of child pornography files. A combination of 
several preliminary considerations must always be ensured to pursue this goal. 
For example, a network could be created that both detects posing and predicts 
certain sexually suggestive objects that describe a pornographic context. In this 
way, the data to be analyzed can be further minimized and the detection of 
incriminated files can be further improved. 
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Abstract. Artificial Intelligence (AI) is becoming an integral part of various 
aspects of human life. However, the successful implementation of AI systems 
poses significant challenges. Delays in the implementation of AI in Germany 
and Europe indicate hurdles, particularly for small and medium-sized enterprises 
(SMEs), which are important drivers of the German economy but also have struc- 
tural disadvantages regarding AI implementation. To ensure that these AI systems 
are designed to meet the needs and expectations of end-users, Human Centered 
Design (HCD) has proven to be a promising approach. This paper aims to iden- 
tify gaps and optimization potentials in the implementation process of AI with 
consideration of the Human Centered Design. The paper is based on existing lite- 
rature and case studies to illustrate the benefits of using HCD in AI development 
and to identify weaknesses and optimization potentials in existing models. The 
paper concludes with recommendations for future research in this area. 


Zusammenfassung. Die kiinstliche Intelligenz (KI) hat sich zunehmend in 
verschiedenen Bereichen des menschlichen Lebens etabliert. Jedoch geht 
die erfolgreiche Integration von KI-Systemen nicht ohne erhebliche Heraus- 
forderungen vonstatten. In Deutschland und Europa wurde die KI-Adoption 
durch Verzögerungen beeinträchtigt, besonders für kleine und mittelständische 
Unternehmen (KMU). Diese KMU bilden eine grundlegende Stütze der deut- 
schen Wirtschaft, sehen sich jedoch spezifischen Hindernissen bei der KI- 
Einführung gegenüber. Ein vielversprechender Ansatz zur effektiven Aus- 
gestaltung von KI-Systemen ist das Human Centered Design (HCD), welches 
den Menschen in den Vordergrund der KI-Entwicklung rückt. Dieser Artikel 
widmet sich der Analyse der KI-Einführung mit Schwerpunkt auf den Prinzi- 
pien des HCD, beleuchtet Defizite in vorhandenen Modellen und unterbreitet 
Empfehlungen für zukünftige Forschungsrichtungen. Das übergeordnete Ziel 
besteht darin, verantwortungsvolle und leistungsfähige KI-Systeme für diverse 
Anwendungsfelder zu fördern. 

Die Implementierung von KI-basierten Assistenzsystemen oder KI-An- 
wendungen in Unternehmen steht häufig vor der Herausforderung, dass Mit- 
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arbeiter mit Skepsis auf derartige Projekte reagieren. Um Akzeptanz und Parti- 
zipation zu steigern, erweist es sich als förderlich, Mitarbeiter bereits im frühen 
Stadium der Konzeption und Implementierung einzubinden. Des Weiteren ist 
es ratsam, einen Verhaltenskodex für die KI-Anwendung zu etablieren, um das 
Vertrauen der Mitarbeiter zu gewinnen. Eine hohe Beteiligung der Mitarbeiter 
geht einher mit gesteigertem Engagement und Akzeptanz. Die Entwicklung 
von Qualifikationen und Kompetenzmodellen muss den Anforderungen des 
digitalen Zeitalters gerecht werden. Eine zentrale Rolle spielen hierbei multi- 
modales Lernen und Prozessunterstützung zur Förderung lebenslangen Ler- 
nens. Reflexionsschleifen tragen zur Steigerung der Lernerfolge und der Mit- 
arbeiterleistung bei. Die Einbindung der Mitarbeiter fördert die Akzeptanz und 
aktive Teilnahme. 

Zusammenfassend betont der Artikel die Bedeutung des HCD und die Not- 
wendigkeit, Mitarbeiter frühzeitig in den Prozess einzubeziehen, um erfolg- 
reiche und akzeptierte KI-Systeme zu gestalten. Er skizziert Empfehlungen 
für die praktische Umsetzung und für künftige Forschungsrichtungen, mit dem 
Ziel, KI-Implementierungen wirkungsvoller zu gestalten und das Potenzial die- 
ser Technologie voll auszuschöpfen. 
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1 Introduction 


Artificial Intelligence (AI) has become increasingly pervasive in various aspects of 
human life, ranging from healthcare and finance to education and entertainment. The 
integration and successful implementation of AI systems, however, present considera- 
ble challenges. In Germany and Europe, the adoption of AI has been hampered by de- 
lays and obstacles, with notable implications for small and medium-sized enterprises 
(SMEs) [1, 39, 16]. These SMEs play a crucial role as drivers of the German economy 
but face distinctive disadvantages in AI implementation. To ensure that AI systems are 
effectively designed to meet the needs of end-users, a promising approach known as 
Human Centered Design (HCD) has emerged. HCD focuses on placing human consi- 
derations at the forefront of AI development, thereby improving the overall user expe- 
rience and system performance. By embracing a human-centered approach, potential 
gaps, and areas for optimization in the implementation process of AI can be identified. 
This paper aims to delve into the realm of AI implementation, with a particular focus 
on the application of HCD principles. Drawing upon existing literature and relevant 
case studies, the paper seeks to illustrate the benefits of adopting HCD in AI develop- 
ment while highlighting weaknesses and optimization potentials within existing mo- 
dels. By analyzing these gaps, it becomes possible to propose recommendations for 
future research in order to enhance the effectiveness and impact of AI implementation 
processes. The subsequent sections of this paper will provide an in-depth exploration 
of the challenges faced in implementing AI systems in Germany and Europe, with a 
specific emphasis on SMEs. Additionally, the principles and practices of HCD will be 
examined to shed light on its value in addressing these challenges. The paper will also 
present case studies to illustrate successful applications of HCD in AI development. 
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Finally, based on the insights gained from the analysis, recommendations will be put 
forth for future research directions in this area. By critically examining the implemen- 
tation of AI and emphasizing the significance of HCD, this paper aims to contribute 
to the advancement of AI technologies that are efficient and aligned with the needs of 
end-users. Ultimately, this will foster the responsible and effective deployment of AI 
systems, leading to widespread benefits and advancements across various domains. 


2 Methodology 


To achieve the objectives of this paper, a systematic literature analysis was underta- 
ken in March of 2023. A forward meta-analysis of papers was carried out, resulting 
in 500+ entries. The search was conducted using specific keywords related to HCD 
and AI implementation, namely “HCD+AI implementation.” By employing this ap- 
proach, a comprehensive collection of relevant scholarly articles, conference papers, 
and other scientific publications was gathered. The search results were then reviewed 
and filtered based on predefined research criteria. A specific backward search was 
carried including the terminology “human focus” and “SME” which narrowed down 
the available literature sources and opened up the research gap. The criteria further- 
more ensured that the selected sources were directly related to the intersection of 
HCD and AI implementation with specific focus on the human and the SME sector, 
providing insights into the challenges and optimization potentials in this domain. In 
terms of ensuring quality and topicality (in-depth papers not older than 5 years) of the 
research, peer reviewed (quality assessment) and highly ranked (topicality) journals 
were used for the in-depth analysis. The intention of this systematic literature analysis 
was to gather a representative sample of the existing body of knowledge on the topic. 
An effort was made to cover a wide range of publications to capture the diversity of 
perspectives and findings in this field. Through the analysis of the selected studies, 
common weaknesses and gaps in existing models were identified. These overlaps in 
weaknesses provided evidence that the identified challenges were not isolated inci- 
dents but rather prevalent across different AI implementation models. This leads to the 
conclusion that the weaknesses identified in the examined studies are likely to exist in 
other models as well. Based on the findings from the literature analysis, recommenda- 
tions were derived to address the identified gaps and optimize the AI implementation 
process. These recommendations serve as valuable insights for future research and 
development efforts in the field of HCD and AI implementation, aiming to enhance 
the design, deployment, and user experience of AI systems. It is important to note that 
this methodology of systematic literature analysis provided a rigorous and structured 
approach to gather and analyze existing scientific knowledge. The inclusion of mul- 
tiple scientific databases and the use of specific search terms ensured a comprehensive 
search process, while the predefined research criteria guided the selection and exami- 
nation of relevant sources. 
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3 State ofthe Art 


HCD is a process-oriented approach, according to Norman [2013], which focuses on 
designing products, services, and systems tailored to the needs and requirements of 
users. The HCD process involves several interconnected steps that may include itera- 
tive loops: 


1. Understanding users: Comprehending the needs, desires and behaviors of users. 
2. Defining problems: Specification of the requirements and constraints that must be 
considered during development. 

. Ideation: Generating ideas to solve the defined problem. 

. Prototyping: Implementing ideas into tangible solutions. 

5. Testing and Feedback: Testing prototypes with users to obtain reactions and feed- 
back. 

6. Refinement: Re-adjusting and adapting concepts based on results from the testing 
step [cf. 36]. 


BW 


The integration of Human-Centered Design into the developing process of AI is the 
result of an increasing focus on designing technology that aligns with user needs and 
preferences. The integration of HCD-approaches into the Al-implementation process 
enables faster and more efficient data processing, as well as generating insights that 
can improve the user experience and support decision-making [cf. 53]. 


3.1 Comparison of Existing Models 


Overall, there are various ways in which HCD and AI can intersect. The choice of ap- 
proach depends on specific requirements, including goals, target audience, and availa- 
ble resources. However, careful integration of AI into the design process can help create 
better products and services that align with the needs and perspectives of users. HCD 
is a crucial approach to ensure that the development and implementation of AI systems 
are aligned with the needs and values of humans. However, there is no one-size-fits-all 
approach for implementing HCD in AI, and various models have been proposed by re- 
searchers and practitioners. This topic aims to compare existing models for HCD in AI- 
implementation, explore their strengths and weaknesses, and identify potential areas for 
improvement. By analyzing and synthesizing these models, optimization potentials can 
be conducted, which can be considered in future research (Table 1). 

This paper mainly focuses on the approach of human-centered AI design. The 
further theories shown are seen as complementary to the primary approach. As men- 
tioned above, implementation strategies of AI projects often lack a human-centered 
approach and neglect the consideration of humans during the implementation process 
[cf. 43; 21]. However, there are SME criteria that should be considered when imple- 
menting AI projects, such as limited resources, limited expertise, and often a lack 
of experience in implementing AI projects [cf. 49]. Although there are good models 
HCD, they are often not implemented, leading to inadequate implementation of AI 
projects. Overall, the models are a useful contribution to the theoretical implementa- 
tion of AI, but the implementation of real use cases is problematic. 
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Table 1. Comparison of different HCD-AI-Approaches 


87 


Approach 


Human-Cente- 
red AI Design 
[cf. 4] 


Description 


refers to integration 
of HCD principles 
into development of 
AI systems; AI sys- 
tems must consider 
the needs, abilities, 
perspectives of users 
to ensure suitability 
for human use; helps 
to make AI systems 
more effective and 
user-friendly 


Strengths 


increased user 
trust and accep- 
tance; improved 
usability and 
accessibility; 
reduced bias and 
discrimination 


Weaknesses 


requires signifi- 
cant expertise and 
resources; possi- 
bility of ethical 
trade-offs 


Optimization 
potential 
integration of 
user feedback 
throughout the 
AI development 
process; ongoing 
evaluation and 
refinement of AI 
systems 


AI-Supported 
Human-Cen- 
tered Design 
[cf. 7] 


assistance in carry- 
ing out tasks within 
scope of HCD; AI 
may help analyze 
user data and iden- 
tify patterns to gain 
a better under-stand- 
ing of user needs; 

AI may aid in gene- 
rating design options 
and prototypes to ac- 
celerate and enhance 
design process 


better efficiency 
and accuracy in 
data analysis; 
increased design 
innovation and 
creativity 


dependent on 
data quality and 
availability; risk 
of over-reliance 
on Al-generated 
insights 


development of 
transparent and 
interpretable AI 
models; collabo- 
ration to ensure 
balance between 
human and AI 
insights 


AI-Driven Hu- 
man-Centered 
Design [cf. 28] 


Al as direct role in 
HCD by being in- 
corporated as part 
of design itself; may 
be used for product/ 
service personal- 
ization and recom- 
mending individual 
customization; may 
improve user expe- 
rience by analyzing 
interactions between 
users and products/ 
services and adjus- 
ting user needs and 
preferences 


increased speed 
and scalability of 
design processes; 
ability to process 
large and complex 
data sets 


possibility of 
reduced human 
input and crea- 
tivity; risk of 
designing for the 
AI rather than for 
the user 


integration of 
user feedback and 
HCD principles 
in the AI develop- 
ment process; 
regular evaluation 
of the impact of 
AI on the design 
process 


(Fortsetzung) 
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Table 1. (Fortsetzung) 


Approach | Description | Strengths Weaknesses | Optimization 
potential 


Ethics-Based | integration of ethical| higher trust and | potential for con- | integration of 
Human-Cente- | considerations into | social responsi- | flicting ethical va- multiple per- 
red Design [cf. | HCD; ensure that AI | bility; alignment | lues and perspec- | spectives and 


25] systems are human- | with regulatory | tives; need for stakeholder 
centered and ethi- | requirements and | expert guidance | input; ongoing 
cally sound; includes} ethical standards | in ethical deci- | evaluation of et- 
consideration of sion-making hical implications 
multiple factors (pri-| throughout the 
vacy, discrimination, | design process 


fairness in develop- 

ment of AI systems; 

| may lead to creation 

of Al systems that 

| are conform to ethi- 
cal standards 


4 Soft and Missing Spots of Human-Centered AI 
Implementation 


The introduction of Al in SMEs by using human-centered methods requires not only 
the involvement of humans in all considerations of technology introduction. Based on 
the indications of previous study, a socio-technical specification sheet proves to be ef- 
fective as a starting point, in which the three dimensions of human-technology-orga- 
nization are considered in the specific implementation project, here AI introduction 
[cf. 21; 40; 44; 50], as a human-centered design of the AI process cannot only focus 
on the consideration of human-technology interaction. The design of work processes 
by managers and organizational specifications must also be considered if AI is to be 
developed and used in a human-centered way [cf. 23]. 


4.1 Technology-Organization-Spots 


The implementation of AI solutions in companies can bring new challenges. Weber 
et al. [cf. 51] assume that organizations are better positioned in this process if they 
have specific resources. These so-called organizational capabilities include AI project 
planning and development, data management, and AI model lifecycle management. 
The development of these abilities in the face of new and systemic challenges leads to 
unforeseeable processes in addition to the consideration of human-centered design in 
AI implementation. A study on user experience and usability among scientific develo- 
pers showed that their orientation in the development of corresponding systems is not 
so much guided by user experience as by management and organizational issues [cf. 
13]. For example, the successful implementation of AI in the education sector is re- 
commended to be accompanied by the cooperation of researchers with local teachers 
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so that organizational factors can be incorporated into the implementation process, 
which can affect the sustainable usability of AI [cf. 6]. Al Ali and Badi conclude that 
AI introductions should be accompanied by organization-wide change management 
processes because they can affect all employees and their work organization [cf. 2]. 
The question here is how technological and organizational processes can be integra- 
ted to design the introduction of AI as a supporting and — in some cases — disruptive 
technology. This presents SMEs in particular with additional challenges. SMEs should 
plan to implement the understanding of the AI implementation process as well as en- 
suring human-centeredness in day-to-day business [cf. 37]. Therefore, the approach 
of using an HCD analysis when implementing AI [cf. 53] is interesting. However, the 
effects of AI use are felt across departments and must be incorporated into business 
processes. Therefore, it is worth considering whether focusing on HCD regarding the 
directly affected users is sufficient or whether the involvement of proprietary stake- 
holders should be considered (e.g., managers, HR developers, [cf. 3]). Although the 
national AI strategy for Germany recommends responsible development and use of 
AI, the use of AI in Germany lags forecasts [cf. 18]. The SMEs predominant in Ger- 
many view the use of new technologies such as AI skeptically and rely on conventio- 
nal technologies, such as rule-based systems [cf. 45]. A human-centered introduction 
of AI in German SMEs must also consider organizational restraints and the usage be- 
havior of previous technologies to ensure a human-centered AI development process. 
However, it is also to be considered whether human-centered factors of the AI imple- 
mentation process, as well as ethical considerations, could be implemented as dangers 
in the life cycle of an AI system [cf. 50]. 


4.2 Human-Technology-Spots 


As stated, it is recommended to design the process of implementing AI with a hu- 
man-centric approach. This also involves the human-centered design of human-ma- 
chine interfaces as the connection between humans and technology in the company. 
For example, the ability to use AI is also a design issue, which can be ensured by 
early and transparent involvement of users in the development and design process of 
Al itself [cf. 31]. On the other hand, in the study of “Participatory Design in the En- 
gineering Design Educational Environment,” user involvement is not participatory in 
development, but tested through experiments afterward [cf. 12]. Technology accep- 
tance also depends on whether the personal needs of users are considered, which 
includes process knowledge about the possibilities of AI and participation in defi- 
ning the problem background [cf. 52; 38]. For example, it is part of the process to 
determine the scope of involvement of a worker together with the work planner and 
production manager for a human-centered introduction of AI [cf. 38]. However, this 
approach in practice faces limitations when workers cannot be released from the com- 
pany’s perspective (e.g., staff shortages) or when conflicts arise regarding the inter- 
pretation and legitimation of human-friendly work design [cf. 9]. The implementation 
of ethical and human-centered design guidelines must be evaluated and negotiated in 
corporate practice. Corresponding guidelines are defined as principles that must be 
translated into the daily work of employees, managers, and executives [cf. 37; 47]. 
Currently, there is still discussion about which competencies are even useful for hu- 
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man-centered Al use, which roles AI and humans take on, and how complex compu- 
tational processes should be taught from a learning theory perspective [cf. 17; 35]. 
Regarding proprietary users (in the work system environment) in the company, such 
as executives and personnel managers, this knowledge is relevant to support work- 
place-related competency development processes. They can only become role models 
for digital transformation in the company if corresponding gaps are closed. 


4.3 Human-Organization-Spots 


However, the need for competency development among members of the organization 
extends not only to AI application knowledge regarding the technology solution to be 
introduced. Rather, the requirements for interdisciplinary skills associated with chan- 
ges in work processes and responsibility structures are also increasing. A new leader- 
ship-, error-, and knowledge culture is considered essential in the AI implementation 
process to make the disruptive technology beneficially used in the company [cf. 21]. 
Regarding the roles negotiated in the socio-technical system between humans and the 
organization, the following requirements arise, which are inadequately considered in 
existing AI implementation models. Competency development is often implemented 
as AI training, but informal learning and creative processes in the team and work- 
place, and thus the integration of the company’s learning process into the AI design 
process, are particularly important for adaptation and fit [cf. 32]. Employees and ma- 
nagers are equally affected by the transformation of the digital working world. For 
managers, it may become necessary to establish a new leadership culture [cf. 19]. 
Herrmann and Pfeiffer speak of organizational embedding of the AI implementation 
process when changed role requirements as well as rules for dealing with AI results 
and constraints must be considered in the everyday work of managers and HRM [cf. 
2022]. In the area of AI implementation, there is increasing discussion about whether 
Al is attributed a social role. The use of ML algorithms as affective computer agents 
that interact with work groups through suggestions and participation in conversations, 
or that can proactively detect emotional and stress situations, is a growing topic within 
the team awareness discussion [cf. 48; 41]. The integration itself as well as the recom- 
mended methodology within the process of integrating employees into the compe- 
tency development process in the context of AI applications is still unclear. The issues 
range from accounting aspects and the question of cost-effective integration of emp- 
loyees to the determination of performance indicators for tracking learning progress. 
Competence development as such remains a permanent topic in organizations, which 
is further reinforced by personnel shortages and lateral entrants. In addition, there is 
a backlog in the development of digital competences, especially in SMEs. In SMEs, 
qualification progress is tracked more manually and therefore not updated daily, and 
soft skills are only partially mapped [cf. 42; 5]. The measurement of knowledge and 
understanding of AI applications, especially in the competency development process 
of employees, is unclear [cf. 41]. Currently it cannot be adequately proven at what 
point an employee has sufficiently understood the AI to be able to apply, operate, and 
monitor it. The operationalization of experiential knowledge is not finally clarified [cf. 
41]. Studies on digital skills in German SMEs also show that digital competencies are 
not being implemented mainly due to lack of time and high costs [cf. 34]. An evalua- 
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tion by the OECD of 100 case studies on AI implementation, however, showed higher 
competency requirements that accompany the use of Al [cf. 33]. In addition, it beco- 
mes clear that the role of informal learning in the AI implementation process is often 
underrepresented, although it offers potential for human-centered adaptation of Al in 
the workplace and in the team. A shared definition of the role of Al in the organization 
and a clear operationalization of the competency development process are necessary 
to successfully implement AI [cf. 32]. In conclusion, the main challenges of imple- 
menting Al lie in digital skills, but also in the acceptance of organizational change. 
According to the HTO (human-technology-organization)-system [cf. 44] from a work 
design perspective, the role of humans remains unclear. This means that the relation- 
ship between humans and the organization is not co-developed — Human-Machine-In- 
terface (human-technology) is considered as well as socio-technical systems (techno- 
logy-organization), but not H-O (human-organization). This is especially problematic 
due to the overload of H-T (human-technology) design research before H-O design 
research, which risks prioritizing technological design over human-centered design, 
contradicting the principles of HCD. 


5 Optimization Potentials and Solution Approaches 


Building on the soft and missing spots in current models, this paper presents solutions 
that are currently being tested in German business practice within a practical AI pro- 
ject. 

The Competence Center for the Future of Work in the “PerspektiveArbeit Lausitz” 
(PAL) project investigates how AI solutions can be implemented in a practice-oriented 
manner. The aim is to introduce data-based assistance systems, including artificial in- 
telligence, in small and medium-sized enterprises in Lusatia in Saxony and Branden- 
burg in order to cope with structural change. This is a practical project because the 
AI solutions are ensured through participatory involvement of teams from four uni- 
versities, 23 companies, and associations. The “PerspektiveArbeit Lausitz” project is 
being funded within the framework of the funding guideline “Future of Work: Regio- 
nal Competence Centers for Labor Research. First round of competition: Design of 
new forms of work through artificial intelligence” in the program “Innovations for the 
production, service and work of tomorrow” by the Federal Ministry of Education and 
Research under the funding number 02L19C306 with a project duration of 01.11.2021 
— 31.10.2026. The Lusatia region is characterized by three major challenges, the so- 
called “3D”: demographic change, decarbonization and digitalization. Demographic 
change is reflected in Lusatia by a sharp decline in population and an aging popula- 
tion. The population of Lusatia fell by around 25% between 1990 and 2019, while 
the proportion of people over 65 rose from 15.5% to 26.1% [cf. 55]. This develop- 
ment also has an impact on the labor market, as there is a threat of labor shortages. 
Decarbonization is another challenge for Lusatia, as the region has traditionally been 
characterized by lignite mining and electricity generation. However, the energy tran- 
sition requires a move away from coal and the expansion of renewable energies. This 
poses major challenges for the economy in the region and requires investments in new 
technologies and infrastructures [cf. 55]. Digitalization is another challenge for Lusa- 
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tia, as the region has so far been less developed in this area than other regions in Ger- 
many. SMEs in the region, in particular, are struggling to meet the demands of digital 
transformation and need to catch up in this area [cf. 54]. These challenges require a 
targeted strategy and investments in the region to maintain competitiveness and create 
sustainable jobs. Based on the challenges presented by the “3D” in the Lusatia region, 
action needs to be taken to address these issues and ensure the region’s competitive- 
ness and sustainability. Deriving from the soft and missing spots in the AI implemen- 
tation models and the experiences from the project PAL, four recommendations for 
action can be derived as to how successful AI implementation can be accompanied. 


5.1 Overall: Reflection Loop 


To ensure effective reflection and learning, it is important to not leave the HTO ref- 
lection loop until the end of a process, but instead reverse the process and incorporate 
reflection as an initial step [cf. 8]. As shown in the PAL-CRISP-DM model, an optimi- 
zation approach of the CRISP-DM-model, it is possible to incorporate reflection loops 
throughout the whole process which meet the first optimization potential (see Fig. 1). 

The introduction of data-based assistance systems or AI applications in companies 
often faces the challenge that employees are skeptical or even critical of such projects. 
In addition to the fear of losing their jobs due to technology, in many cases, techno- 
phobia plays a role, especially among older employees who are afraid of not being 
able to meet the resulting requirements. It is therefore imperative to involve the em- 
ployees directly affected by the project in the conception and implementation of the 
project from the beginning and not only to “take them along” in the sense of “infor- 
ming” them, but to involve them in the implementation of the project and to include 
their know-how. A first prelude for this can be an AI acceptance workshop in which, 
on the one hand, the legal framework conditions (for example, relevant aspects of the 
GDPR) and, on the other hand, the management’s expectations of the project as well 
as a description of the project are discussed with the employees involved. Ideally, this 
workshop is designed to be low-threshold, for example, by first discussing the positive 
aspects of AI applications and examples of using AI solutions in everyday life. 


Feasibility 
discussion 


Division Definition Adaptation 
human- of work of work 
technology tasks processes 


Current state analysis pilot area Target definition Solution development 


Fig. 1 AI procedure model [20, 10] 
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It is also helpful if the company agrees to a code of conduct for the application of 
AI in this context. For example, the “Human Friendly Automation Value Manifesto” 
is a suitable option. Such a code of conduct can visibly communicate the company’s 
approach to the application of AI both internally and externally, thereby providing 
employees and potential applicants with a trustworthy and secure interaction with the 
new technologies. Such an AI workshop should then also be conducted with other em- 
ployees in the next step. This way, uncertainties and resulting unrest in the workforce, 
which becomes aware of the project but has no detailed information about it, can be 
avoided early on. The main message of such a workshop, to which the company ma- 
nagement must also position itself clearly, should be: “We are shaping this process 
together!” 


5.2 Human-Organization: Competence Development Needs Misjudged 


To meet the challenges posed by the changing demands of the digital age, it is ne- 
cessary to develop new qualifications and competence development models. Moving 
away from individual training to the integration of multimodal learning approaches, 
as well as process support, is crucial for employees to remain willing to engage in 
lifelong learning. According to a study by the European Commission, learning in the 
workplace is crucial for employees to adapt to changing requirements and maintain 
their employability [cf. 15]. To achieve success, non-formal and informal learning 
must be planned and structured directly with the help of team processes and reflection 
loops, with iterative reflection loops being a key factor. Research shows that reflection 
loops can be an effective tool for improving learning outcomes and enhancing emp- 
loyee performance [cf. 29]. This involves considering not only technological feasibi- 
lity but also the design of informal learning opportunities. Process thinking is essen- 
tial for employees to move towards lifelong learning. However, digitalization in pub- 
lic administration can be hindered by factors such as process chains from the 1970s, 
which cannot be solved through training. Therefore, process support and project ma- 
nagement are required to help employees overcome such obstacles in both the market 
and the workplace [cf. 24]. Job rotation is a potential solution to help employees adapt 
to changing demands. Studies have shown that job rotation can have a positive im- 
pact on employees’ skill development, job satisfaction, and career advancement [cf. 
46]. Furthermore, the example of introducing Al-assisted systems highlights the im- 
portance of planning the timing of competency development. Although these systems 
can demonstrate their functionality and benefits through use and application, the un- 
derstanding of the processes must be established beforehand. Otherwise, an informed 
decision to participate in the adaptation of the AI solution cannot be presented. This 
means that necessary competence development steps must already be carried out in 
the company before the test phase, so that the employees are empowered sufficiently. 
Research suggests that involving employees in the development and implementation 
of AI solutions can increase their engagement and commitment to the technology [cf. 
26]. Within the framework of the project PAL, a participatory approach is adopted in 
project and process management, with an initial workshop involving employees, ma- 
nagers, and scientists to address any initial questions, create acceptance, and establish 
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initial process understanding, especially for SME [cf. 21]. Including the (learning) ex- 
perience of users in the implementation process addresses all optimization potentials 
of AI approaches. 


5.3 Human-Organization: Termination Criteria and Inclusion 


In the implementation of AI systems, it is of great importance to define termination 
criteria in order to minimize possible errors and risks. Recommendations for discon- 
tinuation criteria can be based on, for example, error rates, trustworthiness of the data 
and the application, ethical considerations, and regulatory requirements. It is import- 
ant to strike a balance between avoiding errors and the benefits of the application. 
Too restrictive an approach can limit the benefits of AI systems, while too open an 
approach increases the risk of errors and harm. Terminations criteria must be defi- 
ned at the beginning of each process. For instance, agile lab teams can be defined, 
that accompany the process on a permanent basis and are responsible for checking 
the termination and success criteria regularly [cf. 19]. This supports transparency and 
collaboration between developers and users, as called for in the second and third op- 
timization potential. The inclusion of employees in the process should also be con- 
sidered in order to achieve a high level of acceptance and participation. This can be 
done through training, workshops, and participatory approaches. A high level of em- 
ployee participation can help identify and address concerns and challenges early on. 
Active involvement can also promote employee acceptance and motivation, which in 
turn supports the success of the implementation. It is important to note, however, that 
employee inclusion should not be seen only as a means to an end. The inclusion of 
employees in the process should be pursued as a goal in its own right, as this can con- 
tribute to a more equitable and inclusive working environment [cf. 11]. 


5.4 Technology-Organization: Further Development of Success 
Indicators 


In addition to Key Performance Indicators (KPIs) that measure the number of loops 
and areas of congestion, it is also important to use satisfaction scores as a measure of 
the success of AI implementation in a company. Customer and employee satisfaction 
are crucial in ensuring the effectiveness of the AI system and can provide valuable 
insights for further improvement [cf. 22]. AI can help improve customer experience 
and satisfaction by providing personalized recommendations, faster response times, 
and more efficient service. This can be measured through customer feedback surveys 
and ratings. Furthermore, AI can help improve employee satisfaction by reducing 
mundane tasks, increasing autonomy, and providing better decision-making support. 
This can be measured through employee feedback surveys and retention rates [cf. 14]. 
Companies should aim to empower themselves to make further adjustments to the 
system without relying on external support. This involves a deep understanding of the 
implemented AI technology and developing internal expertise for system maintenance 
and improvement. As part of the company-specific transformation plan (see Fig. 1), 
satisfaction metrics support the evaluation of the project, which addresses the fourth 
optimization potential 
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6 Conclusion and Outlook 


Considering the theoretical considerations, the view on the implementation figures of 
AI in German SMEs, and the practical application in the PAL project, a human-cente- 
red design for AI implementation can be complemented as following: Human-cente- 
red design considers all persons affected by the product, including executives and per- 
sonnel developers beyond the boundaries of the work system [cf. 9]. Furthermore, the 
connection between HCD and HTO is beneficial for practice, as it reveals gaps and 
solutions in the development of Human-Technology design. However, the practical 
relevance of HCD has not been adequately evaluated. For instance, what constitutes 
sufficient involvement of affected parties? Is feedback from users sufficient, or should 
proprietarily affected parties, such as executives and leaders of other departments, be 
included? Therefore, a participatory approach should be adopted from the outset to 
incorporate practical use cases. Particularly, the neglect of reservations and techno- 
logy experience of executives and management should be considered in AI implemen- 
tation processes. A human-centered AI implementation is always an intervention in 
the prevailing work system and therefore organizational processes. Especially in the 
Competence Centers for Labor Research in Germany, which focus on the design of 
the use of methods and tools of artificial intelligence, there is a chance of human-cen- 
tered design of future work. At the same time, the requirement for a human-centered 
development and introduction of AI requires caution with the obstacles highlighted. 
Especially for SMEs, this offers the opportunity to try out the use of AI and reduce 
the digital divide through AI [cf. 27]. This opportunity is particularly important in 
view of the unclear risk assessments of AI use, in order to open up usage perspectives 
for SMEs [cf. 30]. The strength of the contribution is the consistent HCD along es- 
tablished socio-technical dimensions. The solution approaches of AI-Implementation 
should be further developed [cf. 8]. 
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Abstract. Knowledge Graphs (KG) provide us with a structured, flex- 
ible, transparent, cross-system, and collaborative way of organizing our 
knowledge and data across various domains in society and industrial as 
well as scientific disciplines. KGs surpass any other form of representa- 
tion in terms of effectiveness. However, Knowledge Graph Engineering 
(KGE) requires in-depth experiences of graph structures, web technolo- 
gies, existing models and vocabularies, rule sets, logic, as well as best 
practices. It also demands a significant amount of work. 

Considering the advancements in large language models (LLMs) and 
their interfaces and applications in recent years, we have conducted com- 
prehensive experiments with ChatGPT to explore its potential in sup- 
porting KGE. In this paper, we present a selection of these experiments 
and their results to demonstrate how ChatGPT can assist us in the devel- 
opment and management of KGs. 


Zusammenfassung. Wissensgraphen (englisch Knowledge Graphs, 
KGs), bieten uns eine strukturierte, flexible, transparente, sys- 
temübergreifende und kollaborative Möglichkeit, unser Wissen und 
unsere Daten über verschiedene Bereiche der Gesellschaft und der indus- 
triellen sowie wissenschaftlichen Disziplinen hinweg zu organisieren. KGs 
übertreffen jede andere Form der Repräsentation in Bezug auf die Effek- 
tivität. Die Entwicklung von Wissensgraphen (englisch Knowledge Graph 
Engineering, KGE) erfordert jedoch fundierte Erfahrungen mit Graph- 
strukturen, Webtechnologien, bestehenden Modellen und Vokabularen, 
Regelwerken, Logik sowie Best Practices. Es erfordert auch einen erhe- 
blichen Arbeitsaufwand. 


© Der/die Autor(en) 2024 

C. Zinke-Wehlmann und J. Friedrich (Hrsg.), First Working Conference on Artificial 
Intelligence Development for a Resilient and Sustainable Tomorrow, AIDRST 2023, 
Informatik aktuell, S. 103-115, https: //doi.org/10.1007/978-3-658-43705-3_8 


104 L. P. Meyer et al. 


In Anbetracht der Fortschritte bei groen Sprachmodellen (englisch 
Large Language Modells, LLMs) und ihren Schnittstellen und Anwen- 
dungen in den letzten Jahren haben wir umfassende Experimente mit 
ChatGPT durchgeführt, um sein Potenzial zur Unterstützung von KGE 
zu untersuchen. In diesem Artikel stellen wir eine Auswahl dieser Exper- 
imente und ihre Ergebnisse vor, um zu zeigen, wie ChatGPT uns bei der 
Entwicklung und Verwaltung von KGs unterstützen kann. 


Key words: ChatGPT - knowledge graph engineering - RDF - large 
language model use cases - AI application 


1 Introduction 


In the last years, Artificial Intelligence (AI) has shown great promise in improv- 
ing or revolutionizing various fields of research and practice, including knowl- 
edge engineering. The recent big leap in Al-based assistant chatbots, like Chat- 
GPT (Generative Pre-trained Transformer) model, has created new opportu- 
nities to automate knowledge engineering tasks and reduce the workload on 
human experts. With the growing volume of information in different fields, the 
need for scalable and efficient methods to manage and extract knowledge from 
data that also adapt to new sources is critical. Despite the advances in research 
w.r.t. (semi)automation, knowledge engineering tasks still rely vastly on human 
experts. On one hand, this process can be time-consuming, resource-intensive, 
and susceptible to errors. On the other hand, the reliance on human expertise in 
knowledge engineering exposes it to workforce shortages (as knowledge engineers 
are scarce and the demand is growing) and the risk of expertise loss. These fac- 
tors can impact the resilience and sustainability of systems and operations that 
rely on knowledge engineering. Al-based assistant bot approaches, such as Chat- 
GPT, could bridge this gap by providing a unified tool for tasks in knowledge 
engineering, to reduce the workload of knowledge engineers themselves, but also 
make knowledge engineering more accessible to a broader audience. ChatGPT, 
in particular, has shown promise in generating responses in a variety of syntac- 
tical representations (including code and markup languages) to user queries or 
task descriptions written in natural language. 

In this paper, we discuss and investigate the potential of ChatGPT to sup- 
port or automate various knowledge engineering tasks (e.g. ontology generation, 
SPARQL query generation). We will explore the benefits, pitfalls and challenges 
of using it and identify potential avenues for future research. 


2 Related Work 


ChatGPT, a Large Language Model (LLM) published by OpenAI’, raised the 
interest in the broad field of Machine Learning (ML)? and especially LLMs [4] on 


1 https://openai.com/blog/chatgpt 
? https: //aiindex.stanford.edu/wp-content /uploads/2023/04/HAI_AI-Index- 
Report_2023.pdf 
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a broad scale. While there are current discussions and analysis on the capabilities 
of LLMs like ChatGPT in general (e.g. [1]), there is little in the area of knowledge 
graph engineering. Ekaputra et al. [3] gives a general overview of current research 
on the combination of the broad field of ML and semantic web. 

Searching Google Scholar and Semantic Scholar with “knowledge graph 
ChatGPT”, “ontology ChatGPT” and “rdf ChatGPT” in the beginning of April 
2023 results in only two relevant papers. The first one, [7], reviews the differences 
between conversational AI models, prominent ChatGPT, and state-of-the-art 
question-answering systems for knowledge graphs. In their survey and experi- 
ments, they detect capabilities of their used frameworks but highlight ChatG- 
PTs explainability and robustness. The second one, [6], discusses the usage of 
ChatGPT for database management tasks when tabular schema is expressed in 
a natural language. They conclude among others that ChatGPT is able to assist 
in complex semantic integration and table joins to simplify database manage- 
ment and enhance productivity. The applied approaches and results of these 
two papers indicate that the idea of using LLMs like ChatGPT in the field of 
KG engineering is encouraging and that the LLMs might assist KG engineers in 
their workflows. Still, the research on the usage of LLMs for knowledge graph 
engineers is scarce and seems to be a new research area. 

There exist some non- and semi-scientific resources which render the topic 
from a practical and experience perspective. We want to highlight here a helpful 
blog post by Kurt Cagle [2] on ChatGPT for “knowledge graph workers” and 
a blog post by Konrad Kalicinski [5] on knowledgegraph generation in Neo4J 
assisted by ChatGPT. 


3 LLM-Assisted Knowledge Graph Engineering — 
Potential Application Areas 


In discussion rounds with knowledge graph engineering experts we identified the 
following preliminary list of potential use cases in the domain of knowledge graph 
engineering applicable to LLMs assistance: 


— Assistance in knowledge graph usage: 
e Generate SPARQL queries from natural language questions (related 
experiment in Sect. 4.1 and 4.3) 
e Exploration and summarization of existing knowledge graphs (related 
experiment in Sect. 4.5) 
e Conversion of competency questions to SPARQL queries 
e Code generation or configuration of tool(chain)s for data pipelines 
— Assistance in knowledge graph construction 
e Populating knowledge graphs (related experiment in Sect. 4.4) and vice 
versa 
e Creation or enrichment of knowledge graph schemas / ontologies 
e Get hints for problematic graph design by analysing ChatGPT usages 
problems with a knowledge graph 
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e Semantic search for concepts or properties defined in other already exist- 
ing knowledge graphs 

e Creation and adjustment of knowledge graphs based on competency ques- 
tions 


Given the limited space of this paper, we evaluate a subset of the application 
areas with experiments in the following section. 


4 Experiments 


To evaluate the capabilities of LLMs at the example of ChatGPT for assist- 
ing with knowledge graph engineering, we present several experiments and 
their results. Further details about them is given in the Supplemental Online 
Resources. Most experiments were conducted with ChatGPT with the LLM 
GPT-3.5-turbo® (named ChatGPT-3 from here on), some additionally with 
ChatGPT with the LLM GPT-4* (named ChatGPT-4 from here on). 


:anne a foaf:Person ; foaf:firstName "Anne" ; foaf:surname "Miller" ; 
vcard:hasAddress [ a vcard:Home ; vcard:country-name "UK" ] 

:bob a foaf:Person ; foaf:firstName "Bob" ; foaf:surname "Tanner" ; 
vcard:hasAddress [ a vcard:Home ; vcard:country-name "US" ] 

:wonderOrg a org:Organization . 

:researchDep a org:OrganizationalUnit ; org:unitOf :wonderOrg ; 
rdfs:label "Research Department" . 

:marketingDep a org:OrganizationalUnit ; org:unitOf :wonderOrg ; 
rdfs:label "Marketing Department" . 

:chiefResearchOfficer a org:Role . :marketingManager a org:Role . 

[ a org:Membership ; org:member :anne ; org:organization :researchDep ; 
org:role :chiefResearchOfficer ] 

[ a org:Membership ; org:member :bob ; org:organization :marketingDep ; 
org:role :marketingManager ] 


Listing 1: An organizational KG with two people working in different depart- 
ments of the same organization. 


4.1 SPARQL Query Generation for a Custom Small Knowledge 
Graph 


For a first evaluation, we designed a small knowledge graph as shown in Listing 
1. Specifically, we wanted to know whether (1) GPT can explain connections 
between indirectly related entities, (2) create SPARQL queries over the given 
model and (3) reconstruct the model if all properties and classes were relabelled. 

We issued the following prompt, which includes the knowledge graph from 
Listing 1, on ChatGPT-3 and ChatGPT-4: 


3 https: //platform.openai.com/docs/models/gpt-3-5 
* https: //platform.openai.com/docs/models/gpt-4 
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Prompt 1: Given the RDF/ Turtle model below, are there any connections 
between US and UK? <rdf-model> 


In the knowledge graph of Listing 1, there is a connection between the two coun- 
tries via the two people living in these, which got a job in different departments of 
the same company. While ChatGPT-3 fails to identify this relation, ChatGPT-4 
successfully identifies it in all cases. 

We further asked both ChatGPT models with prompt 2 and received five 
SPARQL queries each, which we analysed for their syntactic correctness, plau- 
sible query structure, and result quality. The results for prompt 2 are listed in 
Tab. 1 and show that both models produce syntactically correct queries, which 
in most cases are plausible and produce corrects results in 3/5 (ChatGPT3) and 
2/5 (ChatGPT4) cases. 


Prompt 2: Given the RDF/Turtle model below, create a SPARQL query 
that lists for every person the country, company and department and role. 
Please adhere strictly to the given model. <rdf-model> 


Table 1. Findings in generated SPARQL queries for prompt 2 


ChatGPT-3 ChatGPT-4 


syntactically correct 5/5 5/5 

plausible query structure 4/5 3/5 
producing correct result 3/5 2/5 

using only defined classes and properties 3/5 4/5 
correct usage of classes and properties 5/5 5/5 
correct prefix for the graph 5/5 4/5 


In essence, Al-based query generation is possible and it can produce valid queries. 
However, the process needs result validation in two dimensions: 1) validating 
the query itself by matching to static information, like available classes and 
properties in the graph, as well as 2) validating the executed query results to 
let ChatGPT generate new queries in case of empty result sets in order to find 
working queries in a try & error approach. 

As a last prompt on the knowledge graph from Listing 1, we created a derived 
RDF graph by relabelling all classes and properties with sequentially numbered 
IRIs in the example namespace, like eg:prop1 and eg:class2. Given the relabelled 
model, we tasked ChatGPT: 
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Prompt 3: Given the RDF/Turtle model below, please replace all prop- 
erties and classes with the most likely standard ones. <rdf-model> 


With ChatGPT-3 only 2/5 iterations succeeded in carrying out all substitu- 
tions. In those succeeding cases, the quality was still not as expected because of 
limited ontology reuse: Only IRIs in the example namespace were introduced, 
rather than reusing the foaf, vcard, and org vocabularies. Yet, the ad-hoc proper- 
ties and classes were reasonably named, such as eg:firstName, eg:countryName or 
eg:departmentName. In contrast, ChatGPT-4 delivered better results: All classes 
and properties were substituted with those from standard vocabularies—foaf, 
veard, and org were correctly identified. For some iterations, ChatGPT-4 used 
the schema.org vocabulary instead of the org vocabulary as an alternative app- 
roach. 


4.2 Token Counts for Knowledge Graphs Schemas 


After the results with the small custom knowledge graph we wanted to check 
the size of some well known knowledge graphs with respect to LLMs. 

The LLMs behind ChatGPT can handle at the moment only 4096 tokens 
(GPT-3.5°) or 8192 respective 32,768 tokens for GPT-4. 


Table 2. Token counts for selected knowledge graphs and serialisations 


Graph Serialisation Type Token Count 
Mondial Oracle DB schema SQL schema 2,608 token 
Mondial RDF schema turtle 5,339 token 
Mondial RDF schema functional syntax 9,696 token 
Mondial RDF schema manchester syntax 11,336 token 


Mondial RDF schema xml/rdf 17,179 token 
Mondial RDF schema json-ld 47,229 token 
Wine Ontology turtle 13,591 token 
Wine Ontology xml/rdf 24,217 token 
Pizza Ontology turtle 5.431 token 
Pizza Ontology xml/rdf 35,331 token 
DBpedia RDF schema turtle 471,251 token 
DBpedia RDF schema xml/rdf 2,338,484 token 


We counted tokens for various public knowledge graphs in different serializa- 
tion formats with the library tiktoken® as recommended for ChatGPT. Tab. 2 


5 https: //github.com/openai/tiktoken 
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lists the token counts for a couple of combinations ordered by token count. More 
data and information is available in the Supplemental Online Resources. The 
turtle serialization seem to result in minimal token count, but is still bigger than 
the similar SQL schema added for comparison. All knowledge graphs exceed the 
token limit for GPT-3.5 and 3 of 4 knowledge graphs listed here exceed the limit 
for GPT-4. 


4.3 SPARQL Query Generation for the Mondial Knowledge Graph 


In addition to the experiments with the small custom knowledge graph (see 
Sect. 4.1) we tested ChatGPT with the bigger mondial knowledge graph? which 
is published since decades with the latest “main revision” 2015. 

We asked ChatGPT to generate a SPARQL query for a natural language 
question from a sparql university lecture’. We used the following prompt five 
times with ChatGPT-3 and ChatGPT-4 each: 


Prompt 4: Please create a sparql query based on the mondial knowledge 
graph for the following question: which river has the most riparian states? 


The results are documented in the Supplemental Online Resources together with 
detailed comments on the given queries. Tab. 3 gives some statistics. In summary, 
all SPARQL queries given by ChatGPT were syntactically correct, but none of 
them worked when executed. Actually all queries had at least one error prevent- 
ing the correct execution like referencing a wrong namespace, wrong usage of 
properties or referencing undefined classes. 


Table 3. Findings in generated sparql queries for prompt 4. 


ChatGPT-3 ChatGPT-4 


syntactically correct 5/5 5/5 

plausible query structure 2/5 4/5 
producing correct result 0/5 0/5 

using only defined classes and properties 1/5 3/5 
correct usage of classes and properties 0/5 3/5 
correct prefix for mondial graph 0/5 1/5 


6 https: //www.dbis.informatik.uni-goettingen.de/Mondial 
T https: //www.dbis.informatik.uni-goettingen.de/Teaching /SWPr-SS20/swpr- 1.pdf 
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4.4 Knowledge Extraction from Fact Sheets 


As an experiment to evaluate knowledge extraction capabilities, we used PDF 
fact sheets of 3D printer specifications from different additive manufacturing 
(AM) vendor websites. The goal is to build a KG about existing 3D printers 
and their type as well as capabilities. We fed plaintext excerpts (extracted via 
pdfplumber) from these PDFs into ChatGPT-3 and prompted it to: 


Prompt 5: Convert the following $$vendor$$ 3d printer specification into 
a JSON_LD formatted Knowledge Graph. The node for this KG should be 
Printer as a main node, Type of 3d printer such as FDM, SLA, and SLS, 
Manufacturer, Material, Applications, and Technique. 


Since the fact sheets are usually formatted using a table scheme, the nature of 
these plain texts is that mostly the printer entity is mentioned in the beginning 
of the text which then is further characterized in a key-value style. As a result, 
the text typically does not use full sentences and contains only one entity that 
is described in detail, but several dependant entities (like printing materials). 
However, the format of the key-value pairs can be noisy. Key names can be 
separated with colons, new line feeds, or in contrast multiple key-value pairs can 
be in the same line, which could impose a challenge. Nevertheless, ChatGPT 
was able to identify the key-value pairs of the evaluation document in a reliably 
way. Unfortunately, it delivered out of 5 test runs for this document 4 partial 
and 1 complete JSON document. In spite of that, we summarize first insights 
gained from a knowledge engineering perspective (but for the sake of brevity, we 
refer to the output documents in the experiment supplements) 


— The JSON-LD output format prioritizes usage of schema.org vocabulary in 
the 5 evaluation runs. This works good for well-known entities and properties 
(e.g. Organization @type for the manufacturer, or the name property), how- 
ever, for the AM-specific feature key names or terms like printer ChatGPT- 
3 invents reasonable but non-existent property names (in the schema.org 
namespace) instead of accurately creating a new namespace or using a dedi- 
cated AM ontology for that purpose. 

— Requesting turtle as output format instead, leads to different results. E.g. 
the property namespace prefix is based on the printer ID and therefore printer 
descriptions are not interoperable and can not be queried in unified way in a 
joint KG. 

— Successfully splitting x, y and z values of the maximum print dimension 
(instead of extracting all dimensions into one string literal) works in 3 runs. 
Although ChatGPT-3 accurately appends the unit of measurement to all x, 
y, z values (which is only mentioned after the z value in the input) in those 
cases, this is a modelling flaw, as querying the KG will be more complex. In 
one run it addressed this issue by separating units into a separate unit code 
field. 
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— A similar effect was observed when it comes to modelling the dependent 
entities. E.g., in 4 runs, the manufacturer was modelled correctly as a separate 
typed entity, in 1 as string literal instead. 


As a general conclusion of the experiment, ChatGPT-3 has overall solid skills to 
extract the key value pairs from the sheets, but the correct modelling or represen- 
tation in terms of a KG significantly varies from run to run. Subsequently, none 
of the generated JSON documents contained sufficient information on their own, 
but only a subset that was modelled accurately. A question for future research is 
whether cherrypicking of individual JSON elements from outputs of several runs 
and combining them into one final document or iteratively refining the output 
by giving ChatGPT generic modelling feedback (like use an ontology, or separate 
unit information, etc.) can be automated in a good and scalable way. 


4.5 Knowledge Graph Exploration 


Experts in the field of knowledge graphs are familiar with concepts from RDF 
Schema (RDFS) (domain/range, subPropertyOf, subClassOf) and Web Ontol- 
ogy Language (OWL) (ObjectProperty, DatatypeProperty, FunctionalProperty, 
...). Often, each of these experts has their preferred tools and methods for gaining 
an overview of an ontology they are not yet familiar with. We asked ChatGPT-3 
two different questions requesting the mermaid® visualization of the most impor- 
tant concepts and their connections: 


Prompt 6: Can you create me a visualization showing the most important 
classes and concepts and how they are linked for dbpedia ontology, serialized 
for mermaid? 


Prompt 7: Can you create me a visualization of the most common con- 
cepts of the DBpedia ontology and their connections focusing on domain 
and range defined in properties. 


We expected a graph with at least eight nodes and their corresponding edges. 
The identifiers for the nodes and edges are expected to follow the Turtle or 
SPARQL prefix:concept notation. If the first question did not achieve the 
goal, we asked additional questions or demands to ChatGPT-3. The results are 
presented in Tab. 4 and we evaluated the displayed graphs based on the following 
criteria: 


8 «a JavaScript-based diagramming and charting tool ...” https://mermaid.js. 


org/. 
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Table 4. Diagram content overview. 


Prompt 6 Prompt 7 
Mermaid Type graph graph“ 
Labels of Nodes prefix and concept prefix and concept** 
Labels of Edges prefix and concept prefix and concept** 
Number of Nodes (total/existing/dbo) 10/10/8 13/12/12 
Number of Edges (total/unique) 12/2 17/17 


* One more prompt was needed to serialize a graph 
** One more prompt was needed to add prefixed labels 


Prompt 6 led to an answer with a hierarchical graph representation of the 
important classes defined in the DBpedia ontology. The diagram already met 
our requirements regarding the minimum number and labelling after the first 
answer and can be seen in the Supplemental Online Resources. 

The class hierarchy was represented by the rdfs :subProperty0f relation, 
and the nodes were labelled in prefix notation, as were the edges. By arranging 
it as a tree using the subClassOf-pattern, only two different properties were used 
for the relations (edges). The root node was of type owl:Thing other nodes are 
connected as (sub)classes from the DBpedia ontology. These were: Place, Organi- 
zation, Event, Work, Species, and Person. The class Work had one more subClas- 
sOf relation to the class MusicalWork. The class Person had the most complex 
representation, with two more subClassOf relations leading to foaf :Person and 
foaf:Agent, the latter of which is a subclass of the root node (owl:Thing). 

In the second prompt (Prompt 7 ChatGPT-3 referred to a graphic file within 
the answer text that no longer existed. Upon further inquiry, a mermaid diagram 
was generated. It was of type “Graph” and contained thirteen common concepts 
and seventeen edges, which were all unique. The labels of both, nodes and edges 
contain no prefixes, but were addable with further inquiry. Only the generated 
concept dbo :Occupation is non-existent. All remaining nodes and edges comply 
with the rules of the ontology, even if the concepts used are derived through fur- 
ther subclass relationships. The resulting diagram is shown in the Supplemental 
Online Resources. 

While prompt 6 leads to a result that can be more comprehensively achieved 
with conventional tools for visualizing RDF, the result from prompt 7 provides an 
overview of concepts (classes) and properties that can be used to relate instances 
of these classes to each other. 


5 Conclusion and Future Work 


From the perspective of a knowledge graph engineer, ChatGPT has demon- 
strated impressive capabilities. It successfully generated knowledge graphs from 
semi-structured textual data, translated natural language questions into syn- 
tactically correct and well-structured SPARQL queries for the given knowl- 
edge graphs, and even generated overview diagrams for large knowledge graph 
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schemas, as outlined in Sect. 4. An detailed analysis revealed that the generated 
results contain mistakes, of which some are subtle. For some use cases, this might 
be harmless and can be tackled with additional validation steps in general, like 
with the metrics we used for SPARQL queries. In general, our conclusion is, that 
one needs to keep in mind ChatGPT’s tendency to hallucinate®, especially when 
applied to the field of knowledge graph engineering where many engineers are 
used to mathematical precision and logic. 

The closed-source nature of ChatGPT challenges scientific research on it in 
two ways: 1. Detailed capability ratings of closed-source probabilistic models 
require much effort 2. Result reproducibility is bound to service availability and 
results might be irreproducible at a later date (due to service changes) 

Thus, open training corpora and LLMS are mandatory for proper scientific 
research. 

In the future, metrics are to be found to rate generated ChatGPT answers 
automatically, like we broached with SPARQL queries. This again enables to 
extend the number of test cases for a specific experiment and to generate pro- 
found statistical results. Another research focus should be given to methods that 
let the LLM access a broader/necessary context to increase the chance for correct 
answers. 
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Supplemental Online Resources 


Details on the experiments described can be found at the following github repo: 
https: / /github.com/AKSW /AI-Tomorrow-2023-KG-ChatGPT-Experiments 
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Abstract. In today’s knowledge society, the necessity for lifelong learn- 
ing can be taken for granted. Hence, everybody will be facing the chal- 
lenge of managing, organizing an optimizing the individual lifelong learn- 
ing journey. Apart from teaching the essential methodological skills and 
abilities to enable lifelong acquisition of knowledge and skills, more and 
more guidance is needed to get an overview of the overabundance of 
learning content from diverse suppliers. 

This paper elaborates on the the basic assumptions, analyses, and 
framework conditions for the development of an “Al-based, Platfor- 
mindipendent cOmpanion-app [for] Lifelong Learning-Optimization” 
(acronym: APOLLO) as part of a 36-month funded project with kind 
financial support from the German Federal Ministry of Education and 
Training (BMBF) under coordination by the German Federal Institute 
for Vocational Education and Training (BIBB) as part of the innovation 
framework program “INVITE”. It outlines the project idea, describes the 
motivation and problem definition reflected in the educational domain as 
well as providing an overview of the current national and international 
state of research. All these aspects have been taken into consideration 
when defining the functional scope of the application. 


Zusammenfassung. In der heutigen Wissensgesellschaft kann die 
Notwendigkeit zur Befahigung zum lebenslangen Lernen als gegeben 
angesehen werden. Abgesehen von der Vermittlung der essen- 
ziellen methodischen Fähig- und Fertigkeiten, um den lebens- 
langen Wissens- und Kenntniserwerb zu ermöglichen, ist aber auch 
immer mehr Lotsenhilfe beim Verschaffen eines Überblicks über 
das überbordende Angebot an Lerninhalten unterschiedlichster Anbi- 
eter nötig. Dieses Paper skizziert die grundlegenden Annahmen, 
Analysen und Rahmenbedingungen für die Entwicklung einer Ai- 
basierten, Plattformübergreifenden cOmpanian-app [für] Lebenslange 
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Lern-Optimierung (Akronym: APOLLO) im Rahmen eines 36-monatigen 
Förderprojektes mit freundlicher Unterstützung des Bundesministeriums 
für Bildung und Forschung (BMBF) und unter Betreuung durch das 
Bundesinstitut für Berufliche Bildung (BIBB) als Teil der Förderricht- 
linie des Innovationswettbewerbs INVITE. Es umreißt die Projektidee, 
schildert die Motivation und Problemstellung im Bildungsbereich sowie 
den aktuellen nationalen und internationalen Forschungsstand, die zur 
detaillierten Spezifikation des Funktionsumgangs herangezogen wurde. 


Keywords: lifelong learning - Al-assisted learning - skill-assessment - 
Artificial Intelligence - learning paths - individual learning path 
recommendation - ESCO - skill profile - taxonomies - recommender 
systems 


1 Brief Description of the Project Idea 


The APOLLO app undesrstands itself as a “digital travel companion for the life- 
long, individual learning journey”. The goal of the project is therefore to provide 
people with an Al-based, intelligent assistant that accompanies them on their 
entire, lifelong education and training path. An individualized, representative 
skills profile created on the basis of data collected over the long term reflects 
the personal competencies and development potential of the users and supports 
them in career planning and development. Lifelong learning thus becomes trans- 
parent, plannable and manageable; users are empowered to use their maximum 
personal potential for professional (personal) development, to consciously plan 
and actively shape their career path in the medium and long term. This enables 
them to react early and proactively to changes in their own life situation or 
unforeseeable socio-economic changes (e.g. digitalization, COVID, ...). Apart 
from the personal AI assistant to be developed, they will be supported by a 
variety of services, such as consulting offers or further training suggestions. The 
human being with his or her individual abilities and skills, hence, returns to 
the focus of the world of employment, without losing the economic interests of 
employers out of sight. 

A consortium consisting of five partner organizations is working on the devel- 
opment of the cross-platform APOLLO-app (iOS, Androing and possibly a 
web interface will be supported). The consortium lead by the Hochschule der 
Bayerischen Wirtschaft (HDBW) consists of the Bildungswerk der Bayerischen 
Wirtschaft (bbw), the Bildungswerk der Baden-Wiirttembergischen Wirtschaft 
(BiWe), the TUV Rheinland Akademie and the Bertelsmann Stiftung. These 
five organizations are combining their expertise in the educational domain, the 
data-treasure of hundreds of thousands of data sets as well as the already exist- 
ing results of other recent (research) projects to create a personalized education 
guide of the future for everybody’s pocket. 
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2 Motivation and Problem Definition 


Lifelong learning is becoming increasingly important in people’s lives. For sev- 
eral decades now, it has been apparent that the required flexibility in everyday 
working life as well as the ability to reorient and adapt to changing working con- 
ditions with newly emerging skill-profiles are becoming increasingly important 
[1]. This development has been accelerated by the rapid innovations in the field 
of digitalization [2]. 

Some researchers even see the (today so common) fixed company affiliation 
— at least in the segment of top performers — disappearing and predict a shift 
to cross-company, project-based collaboration in loose organizational or merely 
contractual structures; permanent employees will work together with hired or 
leased external experts and other bought workforce to solve problems. In this 
agile, volatile and short-lived labor market of the future, however, only those 
who are willing and able to improve themselves further will be able to survive 
[3]. Only employees constantly acquiring and updating the necessary skills and 
capabilities for the respective work environment will be able to keep up and 
succeed [4]. 

In the area of the so-called ‘Individual Contributors (ICCs)’ (highly qualified, 
multi-nationally deployable and globally sought-after top talents), the challenge 
for those concerned will be limited — in this domain, the competition for the 
best experts has already flared up and is commonplace. Fueled by the digital 
transformation, which is bringing about fundamental changes in for many occu- 
pational profiles or even their complete substitution, adaptability, flexibility and 
the will for further development will become important competitive factors on 
the labor market in the future [2]. In the short term, this trend will propagate 
throughout the entire qualification pyramid, and thus also in medium- to low- 
skilled occupational fields. A successfully completed vocational training can and 
will soon no longer provide a sufficient guarantee to remain employable in the 
long run, but still represents the necessary basis for an own gainful employment 
[5]. Therefore, the overriding goal should be to acquire a degree and to keep 
it up to date. Only those who actively continue their education, adapt to new 
situations and continuously develop their skill profile will be able to survive on 
this labor market in the long run [6]. 

Lifelong learning is the consequence. However, not all employees and job 
seekers have internalized this necessity, yet. APOLLO provides approaches to 
address the basic challenges in this context in a timely, efficient and motivating 
way. The target group is to be sensitized for the necessity of lifelong (further) 
education, motivated to interact, presented with the appropriate information 
and individualized content, and the progress moderated and guided accordingly. 
The employee regains the power to act (instead of reacting) — accompanied by a 
personal, artificial and intelligent training assistant, the interest of the individual 
takes center stage and the focus is on securing or optimizing his employability — in 
accordance with individually set career goals [7]. Based on a lifelong growing skill 
profile, this assistant accompanies the employee, advises and coaches; manages 
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and protects personal data; and is a navigation aid in the jungle of Germany-wide 
further education offers. 


2.1 Starting Point: Learning and Adaption are Life Tasks 


Even a completed vocational training and what has been learned through it 
is no longer sufficient to meet the requirements of professional life in the long 
run. Employability throughout the entire working life can only be achieved if 
everyone continuously develops his or her competences according to the chang- 
ing challenges [8]. It should be noted that this personal development is not only 
explicit (in the form of trainings etc.), but also implicit (during the employment 
phase, daily and ‘on-the-job’). The understanding of the importance and neces- 
sity of lifelong learning must be anchored early in education and, hence, become 
a fixed, integral part of school curricula. It must be part of the basic educa- 
tional mission of every school to impart the necessary competencies and skills 
for continuous, self-motivated further education. This is the only way to ensure 
that the imbalance on the labor market is not exacerbated by personal aptitude 
and access to further education (e.g. restricted by social status). Conversely, the 
comprehensive implementation of appropriate measures can simplify access to 
education and training and thus actively contribute to a democratization of the 
labor market. 


2.2 Initial Situation: The Educational Market is Very Differentiated 


According to the German Institute for Adult Education (Deutsches Institut 
für Erwachsenenbildung), there are approximately 25,000 continuing education 
providers. The spectrum ranges from non-profit to commercial, from regional 
to nationwide and international, from micro enterprises to large institutions [9]. 
This is accompanied by a considerable variety of types of offers, certificates and 
degrees as well as different terminology. In principle, this differentiation is a 
great strength, since it allows a very individual response to different demand 
situations. However, it must be ensured that this individual need situation and 
the respective suitable offer also match. There are many regional and supra- 
regional as well as target-group-specific platforms that are intended to remedy 
this situation [10]. However, they have not yet succeeded in matching needs and 
qualification offers to satisfactory extent. 


2.3 Target Group Analysis: Barriers to Participation in Continuing 
Education 


In the context of the target group analysis, various personas including their 
customer journeys were created on the basis of the data provided internally using 
methods from design thinking. This resulted in various person-specific criteria 
and characteristics which were taken into account in the target group analysis. 
In the following, the central aspects of these analyses are briefly presented. 
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When it comes to increasing participation in continuing education, it is 
important to consider the individual barriers to participation in continuing edu- 
cation. There are many aspects to this. Examples include, that the person is 
not aware of a concrete need for further education or that the existing learning 
concepts offered do not seem to fit. Time constraints due to family and/or work 
commitments are obstacles as well as offers that are perceived as too expensive 
and unfulfilled counseling wishes. If, in addition, there is no support (financial or 
otherwise) from the employer or the employment agency, the person often fails 
to identify a suitable educational offer. 


2.4 Target Group Analysis: Motivation for Participation in 
Continuing Education 


The individual benefit is decisive for the participation in further training. There- 
fore, the three most important motives are maintenance qualification (further 
training to maintain the ability to work in one’s own field of activity), adap- 
tation qualification (further training to be able to meet changed or changing 
requirements of one’s own work) and promotion qualification (further training 
to take over other/higher-value tasks) [8]. The aim of the APOLLO project is 
to encourage the target group to take responsibility for their own employability 
and professional development and for lifelong learning. Often, the honest and 
neutral answering of these questions is hampered by a divergence in the self- 
perception and the perception by others of one’s own competencies. A neutral 
assessment of the users’ abilities and skills is — even with the help of selective 
placement tests or other assessment procedures — usually only very inaccurate 
and therefore not satisfactory. This is where APOLLO comes in. The companion 
app suggests suitable training opportunities based on the user’s individual skill 
profile. This Al-based profile, which is generated from a large number of data 
points, accompanies the person throughout his or her entire lifelong education 
and thus provides a realistic basis for matching education and job offers with 
the individual’s qualifications. 

However, end-users are not the only stakeholders who benefit from a net- 
worked, intelligent, AI- and Big Data-driven education solution. Educational 
institutions can use it to continuously optimize their curricula on the basis of 
the resulting competency profiles and constant monitoring of learning progress 
and objectives, and adapt them dynamically and flexibly to the needs of learners 
as well as companies and supra-regional sponsors (employment agencies, etc.). 
In addition, the transparent recording of potential demand situations can lead 
to forward-looking, predictive capacity planning at educational institutions. For 
companies — regardless of their size — the use of the infrastructure makes sense 
on several levels: on the one hand, they can offer incentives for those interested 
in further education to develop into the respective professional field by indicat- 
ing personnel requirements; possibly, partnerships (similar or the same as the 
dual study) can even be concluded at an early stage. In the case of applications, 
neutral, fact-based analyses of the suitability of candidates can be carried out 
on the basis of their skill profiles and strengths and development potentials can 
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be identified, which can then be included in a career development plan at an 
early stage. The skill profiles can also provide a neutral basis for transparent, 
fair, performance-related remuneration and salary components. 


2.5 Methodological Approach and Data Basis 


For the development of the APOLLO application solution, a modern, agile, 
SCRUM-based development process is followed. The development activity is 
divided into several phases, each of which focuses on one aspect of the APOLLO 
solution. In line with the agile development philosophy andm the design thinking 
methodology, this phase follows a user-centric paradigm: For APOLLO, the max- 
imization of the user benefit is always predominant. This is illustrated, among 
other things, by the modeling of a user journey, which — based on a persona 
analysis — captures the requirements, desires and user processes of the indi- 
vidual stakeholders. From the beginning, close cooperation with the first pilot 
customers was essential to identify all diverse requirements as early as possible. 
It is tried to ensure the basic functionality of the application in a conventional, 
proven way according to the established principles of computer science. In later 
iterations, smart experimental approaches based on artificial intelligence and 
machine learning will be successively added. 

For the modeling and mapping of the skill profiles, the consortium can rely 
on the expertise and work of the Bertelsmann Foundation. Therefore, the imple- 
mentation will be done in close coordination and under consideration of the 
ESCO framework as well as the ONET skill taxonomies. So far, the mapping 
of the skill profiles in a graph database is planned. User skills and abilities are 
also stored in the knowledgebase and enriched with the corresponding ontologies 
according to the ESCO definition. From the attached architecture and concept 
visualizations the preliminarily assumed structure and the corresponding data 
flow can be taken: The user’s skill profile is created (as automatically as possible) 
from different sources (credentials, certificates, self- and third-party assessments, 
...). One challenge will certainly be in the recording of non-standardized compe- 
tencies. (e.g., job references and letters of recommendation in prose), since the 
contents of these must first be extracted, analyzed, and classified by complex, 
AI-based methods (e.g., natural language processing). An elementary part of the 
data structure is therefore the representation of the goals as well as the possibility 
to categorize the user by his parameters (skill profile, goals, etc.) and to derive, 
monitor and control his learning journey based on this data. The required data 
is used to give a weighting to the entities (skills) or nodes. The requirements 
foresee the validation of credentials and/or certificates via blockchain by the 
education providers. These also receive a weighting and are reliable data points, 
especially when comparing and applying analytical procedures from statistics. 
From the weighted references of the knowledgebase, the digital image (quasi a 
digital twin) of the user or his competencies, referred to as user knowledge, is 
created. This represents the core of the user profile. 


Foundations for the Development of an Al-based, Platformindipendent ... 123 


3 Analysis of the National and Intern. State of Research 


The analysis of the national and international state of research leads to a mixed 
picture. In the business environment, the use of skill profiles has become wide- 
spread, especially in larger companies. They are usually used as a proven tool 
in the context of the annual performance evaluation (also as a basis for the 
determination of personal goal achievement), for the identification of develop- 
ment potential and as a basis for the trans- parent description and filling of 
vacant positions in the company [11]. However, the tools currently in use (e.g., 
from SAP, Persis, or Evidenz) only rely on manually created and maintained 
skill profiles, and the assignment of employees to career paths, the monitoring 
of individual development perspectives, and the implementation of training and 
development measures are also performed by the respective HR consultant [12]. 
AI support, the use of big data analytics or process automation is — at best 
— rudimentarily indicated but by no means widely available. There is also no 
networking with cross-company or external training offers and platforms. In the 
educational environment (e.g. at colleges and unive)rsites, however, skill profiles 
are almost unknown, especially in the German-speaking area — apart from the 
rudimentary use in the allocation of university places for courses of study with 
restricted admission. Especially in view of the aforementioned widespread use in 
the professional environment, this is alarming, since the participants in the edu- 
cational system can neither benefit from the use in the context of per- sonalized 
education and training nor are they prepared for the use in the company. Nor 
is there any sensitization about the necessity of or preparation for later lifelong 
learning [13]. 

Only in the university domain of (automated) study support / student coun- 
selling simple standard questions (‘Are you more interested in economics or in 
natural sciences?’) are used in isolated systems, although corresponding solution 
approaches, e.g. for the automation of first- and second-level support in customer 
hotlines, have already been tested millions of times. The fact that these prob- 
lems are often assigned as selective bachelor and master theses, later piloted as 
rudimentary solutions and not professionally implemented as part of an overall 
concept, certainly does not contribute to an increase in quality. 

Whereas, the necessary basic technologies are available, already tested, 
robust and suitable for productive use — but there is a lack of definition of 
an adequate overall package taking into account the requirements and including 
the necessary interdisciplinary , domain-specific knowledge about the education 
and training area. 

In international comparison, there exist — in the university environment — a 
manageable number of positive case studies that address some of these aspects: 
At Georgia Tech University in the USA, students are already successfully sup- 
ported and taught by ‘Jill’, a virtual tutor [14]. The quality of the virtual learning 
environment is so good that students can sometimes no longer tell the difference 
between it and conventional lecturer support [14]. Genie’, a virtual companion 
at Deakin University in Australia, even goes one step further and offers users 
interactive help in designing curicula, selecting additional modules and individu- 
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alized, interest-based scheduling of intra- and extrauniversity activities [15],[16] 
[17]. Proven, existing approaches from the industrial environment can be trans- 
ferred to the entire lifelong training path in a goal-oriented and methodologically- 
didactically sound way. This promises better individual support, transparent 
insights into and full control over their competence profile as well as the result- 
ing possibility to create personalized learning paths and training roadmaps. 


4 Description od Specific App Idea 


The aim of the APOLLO project is to accompany people on their education 
and training path and to support them in the selection of individually tailored 
training measures as well as in the organization of their lifelong learning. The 
individual and his or her needs are at the center of our efforts. People interested 
in further education should be enabled to regain control over their personal 
development, thereby strengthened in their role as employees and enabled to 
reflect and develop their personal strengths and interests as fully as possible in 
a professional profile. It is paramount that the user always has full control and 
decision-making power over his data and its use. In case of doubt, all interactive 
components must always act in the sense of data protection and for (instead 
of against) the user. But a win-win-win situation must also be created for the 
employing companies as well as for the providers and promoters of vocational and 
further training. The project is therefore divided into three main components. 

The definition and development of individualized skill profiles - Al-supported 
and with the help of available data (Big-Data) — for the transparent mapping of 
the abilities and skills of the respective user. They serve, among other things, as a 
basis for the proposal of suitable further training measures in order to achieve the 
goals set by the user or as support for the assessment of the personal suitability 
for the application for advertised jobs. The assistant (which is available to the 
users e.g. on mobile devices in the form of an app) accompanies the individual 
on his lifelong educational journey and represents the personification of the user 
interface. Depending on the preference, the Learning and Skill Companion (LSC) 
offers a target-group-appropriate approach (among other things with the help 
of the methods of serious gamification) and helps in the search and selection of 
further education offers, motivates in the implementation and helps to keep an 
eye on one’s own progress. 

The collected data holds great potential for the participating employers, the 
providers of training and further education as well as the supporting and sup- 
porting authorities. Focal points of interest of those interested in further edu- 
cation as well as the needs of the companies can be identified and responded 
to quickly. Similarly, companies can obtain a better overview of the person- 
nel resources available on the labor market and their skill profiles; based on 
this, forward-looking personnel planning with corresponding roadmaps can be 
defined, bottlenecks can be better avoided and, ideally, investments can be made 
in corresponding personnel development or further training programs and part- 
nerships at an early stage. Both variants represent a decisive added value for the 
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employer side, because in many cases the companies do not have an adequate 
overview of the competence distribution of their own employees. Conversely, 
emerging changes in requirement profiles and job profiles can also be taken into 
account in the planning of curricula and training contents at an early stage, so 
that the (further) training managers can also act predictively and with foresight, 
plan capacities and adapt contents. 


5 Conclusion and Outlook 


Lifelong learning is a cornerstone of individual and personal development. In an 
increasingly fast-paced world, where innovation cycles shorten and technological 
progress sparks disruptive innovations manifesting on Megatrend-scale, human- 
ity is affected globally and on all different levels of society [18]. When technolo- 
gies like the evolution of mainstream information technology, the Internet, the 
(industrial) metaverse, robotics, automation, big data and artificial intelligence 
in all its forms of manifestation start permeating the human living realm, the 
implicitly affect the workspace. Skill-profiles are altered; new skills are required — 
sometimes even over night — and affect the whole workforce, often in very unex- 
pected ways [19]. This does not solely apply to technological disruption, though 
it is usually the fastest, most agile driver of changing skill requirements on the 
labor market. But also other Megatrends, that affect societies on socio-political, 
socio-cultural, ethical-moral and socio-economic level, carry these properties. 
They can act as accelerators as well as inhibitors for societal change [20] [21] — 
but nevertheless this is what they bring: transformation. 

Motivated by these fundamental insights, important interim results have 
already been achieved within the past 20 months of development, including suc- 
cessful alpha testing as part of the ‘BIBB INVITE Tool Check’. With the help of 
several development iterations, it was possible to develop a successfully running 
prototype, which is still being improved and adapted. The basic assumptions 
presented in this paper can still be regarded as valid, and the various projects 
within the INVITE project have produced similar interim results. Due to the 
necessity of lifelong learning, there are still relevant questions for the APOLLO 
project. Therefore, it is necessary to generate the relevant data with the help of 
the research project and to put them into context. 

However, some challenges still lie in the technical depth: The definition of 
suitable, accurate skill profiles is not only a concern of the consortium publishing 
here. The disruptive developments in the field of AI algorithms in recent weeks 
and months — especially in the area of transformer-based language models and 
the associated analysis and generation capabilities in natural language interac- 
tions — shed new light on the technological core of proposal generation. These 
aspects are the subject of ongoing, continued research in the consortium and will 
be addressed within further publications and papers. 
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Abstract. Digital innovations and technologies, particularly artificial intelli- 
gence, offer a unique opportunity to fundamentally transform business proces- 
ses. Small and medium-sized enterprises (SMEs), having a significant impact 
on the German economy, are encouraged to fully embrace this opportunity for 
digital transformation. 

Inspired by the usage of knowledge management as a mediation mechanism 
for effective AI application in [5], this case study examines its practical implica- 
tions. Wiewald, an SME specializing in compressor systems, serves as an appli- 
cation partner in the KMI project, exploring the implementation of AI in SMEs. 

By comparing academic literature on Industry 4.0’s impact on knowledge 
management with industry experts’ perspectives, we develop an appropriate di- 
gitalization strategy based on knowledge management. Its potential implemen- 
tation is discussed using Wiewald as a practical example. 


Zusammenfassung. Diese umfassende Fallstudie taucht in das Gebiet des 
Wissensmanagements (KM) ein und dessen Anwendung bei Wiewald, einem 
Unternehmen, das sich auf die Planung und Gestaltung von Kompressor- 
systemen spezialisiert hat. Die Studie präsentiert eine umfassende Unter- 
suchung verschiedener Dimensionen des KM und bietet strategische Empfeh- 
lungen, die auf Wiewald’s Digitalisierungsstrategie zugeschnitten sind. 

Die Studie hebt die Bedeutung des KM im digitalen Zeitalter hervor, in 
dem die transformative Kraft der digitalen Revolution einen Paradigmen- 
wechsel in der Fertigungsbranche herbeigeführt hat. Das Aufkommen des 
Internet-of-Things und der Zustrom von Massendaten haben den Übergang von 
Big-Data zu Smart-Data erforderlich gemacht, welche handlungsorientierten 
und aussagekräftigen Informationen repräsentiert. Es wird jedoch darauf hin- 
gewiesen, dass Organisationen das volle Potenzial von Smart Data möglicher- 
weise noch nicht vollständig ausgeschöpft haben. 

Die Fallstudie zieht Erkenntnisse aus empirischer Forschung mit malay- 
sischen Fertigungsunternehmen heran und hebt die entscheidende Rolle der 
Integration des KM in Geschäftsprozessen hervor. Vier Dimensionen des KM 
werden identifiziert: Wissensschaffung und -erwerb, Wissensaustausch und 
-transfer, Wissensspeicherung und -ab- frage sowie Wissensanwendung. Diese 
Dimensionen sind miteinander verbunden und betonen die Bedeutung der Nut- 
zung organisatorischen Wissens für effektive Innovationen. 
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Die akademische Literatur zum KM wird durch eine bibliometrische Ana- 
lyse erforscht, um Lücken zu identifizieren und aktuelle Trends sowie auf- 
kommende Techno- logien zu integrieren. Die Studie betont die Notwendig- 
keit, die Kluft zwischen praktischen Leitlinien für die Wissensanwendung in 
der Industrie und dem Stand des Wissensmanagements in der akademischen 
Forschung zu überbrücken. 

In Zusammenarbeit mit Wiewald konzentriert sich die Studie auf die Ent- 
wicklung eines Kompressorsystem-Konfigurators, der dem Cheftechniker bei 
der Planung und Gestaltung neuer Systeme helfen soll. Durch Machbarkeits- 
tests unter Verwendung historischer Daten wurden in der Vergangenheit Ent- 
scheidungsbaummodelle entwickelt, die basierend auf grundlegenden An- 
forderungen wahrscheinliche Entscheidungswege bestimmen können. Dieser 
Konfigurator zielt darauf ab, Vorschläge für gültige Systeme zu liefern und 
schnelle Reaktionen sowie hochwertige Entscheidungsfindung zu ermöglichen. 

Für die Implementierung des Konfigurators wird ein Ansatz namens „Ac- 
tionable Cognitive Twin“ (ACT) vorgeschlagen, der die Transparenz von In- 
formationen, die Entscheidungsunterstützung und die Agilität verbessern soll, 
während er gleichzeitig die Beteiligung der Mitarbeiter an der Gestaltung 
von Geschäftsprozessen und die Förderung von Innovationen anregt. Die Im- 
plementierung von ACT umfasst die systematische Verwaltung von Wissen, 
die Integration von Ontologien und Wissensgraphen sowie den Einsatz daten- 
getriebener Techniken wie Natural Language Processing und neuronalen Netz- 
werken. 


Keywords: Knowledge Management - Digital Transformation - Knowledge 
Application 


1 Background 


Digital innovations and technologies, especially artificial intelligence, present a uni- 
que opportunity to fundamentally change how organizations work: improving know- 
ledge access, facilitating more extensive product and service development, minimi- 
zing bureaucratic obstacles and enabling the emergence of new business domains. 
Put simply, artificial intelligence plays a key role in facilitating a digital revolution 
towards a knowledge-driven economy. 

Small- and medium-sized enterprises (SMEs) play a vital role in the German eco- 
nomy, accounting for 99.4 percent of the market and driving national competence 
development [1, 2]. Fully capitalizing on this opportunity for digital transformation 
is crucial given their significant influence, which is why the German government’s 
strategic initiative for Industry 4.0 [4], aims to leverage and maximize SMEs’ trans- 
formative potential. It is reasonable to assume that by embracing Industry 4.0, SMEs 
can ensure long-term competitiveness in the evolving business landscape, positioning 
themselves for sustained success. 

Unfortunately [3], “Let There Be Change’, sheds light on the limited AI imple- 
mentation among German companies. The study reveals that a mere 12 percent of 
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these companies consider their AI maturity level sufficient to improve their proces- 
ses and organizations. In contrast, a substantial 63 percent remain in the experimental 
phase of adopting Al at an enterprise level, indicating their challenges in leveraging 
Al effectively. 

The empirical study by [5] with 120 senior executives from Italian manufacturing 
firms highlights the crucial role of knowledge management practices (KMPs) in achie- 
ving AI maturity and leveraging its potential. The findings show that AI deployment 
positively influences KMPs, stimulating their higher levels while enhancing supply 
chain resilience (SCR) and manufacturing firm performance (MFP). This emphasizes 
the need for an effective digitalization strategy that prioritizes KMPs as a mediating 
mechanism. By leveraging KMPs to transform data into valuable knowledge, organi- 
zations can enhance decision-making and improve MFP and SCR outcomes. 

The KMI project (“Künstliche Menschlich Intelligent” — federal grant no. 02L 
19C500) champions sustainable AI; prioritizing worker benefits over profit-driven ap- 
proaches like Industry 4.0. By augmenting worker capabilities, reducing menial tasks, 
and ensuring usability, sustainable AI aims to preserve employability, fostering a more 
worker-centred “humane” approach within the Industry 5.0 context, where worker be- 
nefits are valued alongside company profits [4]. 

Building upon the insights of [5], we explore the potential applicability of their 
findings in a real-life SME. Wiewald, a specialist in configuring, installing, and main- 
taining compressor systems, serves as our focus for this investigation. As an active ap- 
plication partner in the KMI project, our research revolves around enabling the sustai- 
nable integration of AI within SMEs, drawing from our own insights and experiences 
gained through our work with Wiewald, which have shed light on additional challen- 
ges commonly faced by SMEs in the project. 

One such challenge is an insufficient ‘data-mindedness’, referring to the unaware- 
ness among SMEs about the opportunity to generate additional value from their data 
for various reasons. In the case of Wiewald, this could be attributed to unstructured 
data or the absence of digitization. Another challenge is the presumed absence of a 
structured and integrated process to externalize and document employee knowledge, 
particularly when tacit expert knowledge is required for projects involving new inno- 
vations such as Al. 

Concerning these two issues, our experience at Wiewald and a new understan- 
ding concerning the importance of knowledge management practices for digital trans- 
formation, we believe that companies like Wiewald, need to rely on knowledge and its 
exploitation, to sustain a long-term advantage. While extensive knowledge manage- 
ment practices may not be essential for day-to-day operations, SMEs must proactively 
prepare themselves with a digitalization strategy to remain relevant in the evolving 
landscape of the German economy’s digital revolution. This strategy ensures their 
long- term success in an emerging knowledge-based economy, where the ability to 
leverage knowledge and adapt to digital advancements becomes a crucial competitive 
advantage. 
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2 Hypothesis and Research Question 


We anticipate that an improved availability and accessibility of data and information 
would concurrently improve the potential for value generation and MFP of SMEs, 
thereby creating the necessary foundation for digital transformation strategies. Hence, 
we hypothesize that a viable solution must include: 


e A means for systematic knowledge management to capture explicit knowledge and 
preserve tacit expert knowledge within the company, explicitly integrating it into a 
business process model. 

e A holistic measure for innovation management that encourages active involvement 
from workers, especially when structuring or modelling business data and business 
processes, ideally facilitating worker-driven initiatives for innovation management. 


By reconsidering our approach regarding a successful digital transformation strategy 
in SMEs, an argument concerning the merit and strategic importance of knowledge 
management practices for the sustainable implementation of AI should be raised and 
evaluated, leading to the following research question: 

“To what extent can the effective and decisive application of knowledge raise an 
SME’s AI maturity level and determine the success of its digital transformation?” 

Evaluating this research question involves assessing the impact of effective know- 
ledge application on AI maturity and its correlation with the success of digital trans- 
formation efforts in SMEs. The examination will consider factors such as the percen- 
tage of explicitly captured business process knowledge compared to processes lacking 
explicit detailed information, as well as tracking the number of adjustments and inno- 
vations introduced by workers since the inception of the digitalization solution, along 
with their impact on MFP. 

This research question delves into the relationship between knowledge manage- 
ment practices (KMPs), AI maturity, and the success of digital transformations in 
SMEs. It seeks to demonstrate the crucial role of KMPs in elevating an SME’s AI ma- 
turity level and achieving a successful digital transformation. 

To address this research question, the case-study will provide a concise and com- 
prehensive overview of knowledge management, exploring its various dimensions, 
inter- connections, and their impact on driving process and product innovation. By 
doing so, it will examine how effective knowledge management enhances AI matu- 
rity within organizations and facilitates the development of impactful digital trans- 
formation strategies. 


3 Theory: Global Perspective - Comparing Industry 
Experience with Structure of Academic Research 


The digital revolution, driven by advancements in technology such as the Internet of 
Things, has brought about a transformative shift in the manufacturing domain [7]. It 
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has enabled seamless communication and autonomous collaboration among machi- 
nes, leading to a massive influx of data commonly known as big data. However, it’s 
important to note that big data is not synonymous with smart data, which represents 
actionable and meaningful information. Despite the prolonged existence of the digital 
revolution, there remains a possibility that organizations have not fully transitioned 
towards harnessing the potential of smart data [10]. 

As organizations are encouraged to adapt to the emerging knowledge-driven eco- 
nomy, investigating the transformation from big data to smart data becomes essential. 
This investigation involves examining the differences and similarities between con- 
temporary theoretical perspectives and practical experiences, providing insights into 
the evolving role of knowledge management in the digital era. By exploring this in- 
terplay, we can comprehensively understand how the digital revolution has influenced 
knowledge processes within manufacturing companies, thus establishing the theoreti- 
cal framework to effectively address our research question. 

In their empirical work, [6] conducted a questionnaire involving 162 Malaysian 
manufacturing firms, highlighting the significance of integrating knowledge manage- 
ment (KM) into business processes. They concluded that KM plays a crucial role in 
driving process- and product innovation, leading to innovative value creation. Their 
findings provide valuable insights and guidelines for effective KMPs. 

The study defines four dimensions of KM (shown in Fig. 1), which serve as the 
foundation for our case study. These dimensions examine how organizational know- 
ledge is: created and acquired; shared and transferred; stored and retrieved; and ap- 
plied. Their findings emphasize the importance of interrelationships among KM di- 
mensions for their overall effectiveness, increasing the firm’s potential to leverage the 
inherent value of their data for effective innovations [6]. 


Knowledge 
Acquisition 


a Knowledge 


Technological 
Innovation 


Knowledge 
Application 


Knowledge 
Storage 


Fig. 1 Overview of the 4 knowledge management dimensions, their interrelationships and 
influence on technological innovation, according to [6] 
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[7] contributes to the prior industry experience by providing their academic per- 
spective. They conduct a bibliometric analysis of 90 relevant articles and use cluster 
analysis to explore the intellectual structure of academic literature related to know- 
ledge management in Industry 4.0. Their analysis results in the identification of six 
clusters of relevant keywords, which serve as the basis for their systematic literature 
review. Their research allows them to identify gaps in the existing academic litera- 
ture and integrate current trends and emerging technologies, ideas, and concepts. This 
enables them to provide insights into the future directions that the knowledge manage- 
ment literature may take. 

Therefore, by comparing the practical guidelines for applying knowledge manage- 
ment in industry with the state of knowledge management in academic research, it is 
possible to integrate emerging concepts and ideas related to KM. This integration all- 
ows for the identification of parameters that contribute to the development of a robust 
digitalization strategy applicable to SMEs in general, and specifically to Wiewald. 


3.1 Knowledge Acquisition & -Creation 


The academic sphere places significant emphasis on ‘knowledge creation’, with ap- 
proximately 50% of the papers in the dataset addressing this topic [7]. The creation 
of knowledge involves the generation of new knowledge within the company or its 
acquisition from external sources. By adopting this broader definition of knowledge 
acquisition, its link to innovation is reinforced, as the creation of knowledge signifi- 
cantly enhances innovative performance [6]. Consequently, when internal and external 
knowledge sources are effectively combined, the acquisition of knowledge exerts a 
positive influence on innovation. 

For this case study, ‘knowledge acquisition’ involves discovering, locating, crea- 
ting, or capturing new information, including both intangible tacit knowledge and tan- 
gible explicit knowledge from diverse sources such as employees, data, and external 
entities [6, 7]. [7] primarily focusses on “knowledge capture,’ highlighting its positive 
relationship with ‘knowledge sharing’ and ‘-transfer’, with guidelines promoting an 
effective knowledge-sharing culture that is impactful to decision-making and thereby 
relevant to the ‘knowledge application’ dimension [6]. 

‘Knowledge creation’ trends overlap with ‘-transferability’ trends, such as smart 
and digital environments. This results in two sub-streams of academia; process model- 
ling and condition monitoring, which prioritize cyber-physical- and semantic techno- 
logies like digital twins, semantic webs, and semantic web rule languages to enhance 
‘knowledge acquisition’ capability [7]. 


3.2 Knowledge Sharing 


‘Knowledge sharing’, also known as ‘knowledge accessibility, allows employees to 
share and access both tacit- and explicit knowledge within and outside the organiza- 
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tion [7]. The presence of a culture of ‘data-mindedness’ acts as a moderating variable, 
enhancing the speed of information flow, thereby emphasizing the need for an integra- 
ted transparent knowledge-sharing behaviour [6]. 

This transparency creates a ‘proximity effect’, enabling increased access to infor- 
mation and faster knowledge dissemination during day-to-day operations [6]. Contem 
porary views highlight the paramount importance of ‘knowledge sharing’, evident in 
its prevalence across academic research and its influence on various trends [7]. 

Although the transfer of knowledge is often regarded as the most significant 
knowledge management dimension [6], its direct impact on strategic objectives, such 
as achieving higher Al-maturity, may be debatable. 


3.3 Knowledge Storage 


‘Knowledge storage’, or ‘knowledge documentation,’ is vital in the knowledge ma- 
nagement process, involving capturing, refining, structuring, integrating, and storing 
information [6]. This concept resonates with academic topics such as big data, digital 
transformation, and cyber-physical systems that are heavily influenced by the digital 
revolution [7]. 

The concept of ‘organizational memory’ involves capturing tacit knowledge and 
enhancing its accessibility, thereby preventing the loss of valuable information [8]. 
This, in turn, fosters ‘organizational agility’ by enabling organizations to assimilate 
up-to- date knowledge and respond swiftly to new information. By combining these 
practices with the ‘proximity effect’ and fostering a culture of ‘data-mindedness’, 
knowledge storage practices accelerate the transmission and dissemination of know- 
ledge, thereby driving innovative behaviour [6]. 

It is reasonable to assume that within such a dynamic environment, shaped by ef- 
fective knowledge storage, that organizational knowledge can be leveraged to make 
well- informed decisions for long-term sustainable growth and an immediate compe- 
titive edge. 


3.4 Knowledge Application 


In the context of the global paradigm shift driven by the digital revolution, the import- 
ance of ‘knowledge application’, also known as ‘knowledge responsiveness’, becomes 
increasingly pronounced [6]. It represents a strategic asset for knowledge-driven orga- 
nizations, facilitating a realignment of goals towards long-term strategic value crea- 
tion. This realignment extends beyond the scope of business intelligence; which focu- 
ses on immediate data utilization, and enters the realm of data science. Through data 
science, organizations can generate profound insights that drive innovative advance- 
ments in business processes and products. Beyond supporting organizational goals, 
such strategic utilization of knowledge promotes collaboration and innovation, leading 
to the development of new products and processes. In fact, research has shown that 
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effective knowledge application facilitates the translation of organizational expertise 
into innovative advancements, contributing to overall organizational innovation [6]. 

This transformative process towards advanced data-driven decision-making, in- 
cludes the concept of ‘responsiveness’ that encompasses both ‘agility’ in reacting to 
changes in information and the ‘quality of response’; which empowers organizations 
to take strategic actions and unlock new possibilities of higher quality, agility and ef- 
fectiveness [6]. 

While the Malaysian industry questionnaire recognizes the strategic significance 
of ‘knowledge application’, ranking it as the second most important dimension in re- 
lation to innovation [6], the academic literature on KM reveals that it is the least in- 
vestigated dimension, with only 16% of the papers in the dataset referencing know- 
ledge application [7]. This discrepancy underscores the need to address this potential 
research gap and explore how the effective application of knowledge can bridge the 
gap between information advantages and actionable decisions. 

Drawing insights from the investigative results of [5] and [7], we align our under- 
standing with the current state of knowledge management practices (KMPs) in the 
field. These results emphasize the vital role of ‘knowledge application’ in achieving 
impactful outcomes in the digital era [3, 5]. It becomes evident that SMEs need to 
prioritize a digitalization strategy that effectively applies their knowledge resources to 
create a competitive advantage and foster innovation. By doing so, they can thrive in 
the rapidly evolving digital landscape. These findings underscore the significance of 
strategic knowledge management and its relevance in driving success for SMEs in the 
digital age. 


4 Implication and Conclusive Recommendation for Wiewald 


In partnership with Wiewald, we have been working on a compressor system ‘confi- 
gurator’, that is supposed to assist their chief technician during the planning and de- 
sign of new compressor systems. With early feasibility tests using historic data that 
was available to us at the time, we were able to design simple decision tree models 
that were able to determine most likely decision paths, when given basic requirements 
about desired compressor capacities. This initial work has sharpened our vision for 
the digital solution that suits Wiewald’s requirements. 

As the chief technician elicits the requirements from their customer, he enters the 
relevant conversation data into his electronic quotation form. The configurator uses 
this input to perform calculations in a discrete backend service. Once the basic data- 
points are determined, the configurator should be able to provide suggestions for valid 
compressor systems. 
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Fig. 2 Approximation of the current proposal creation process at Wiewald 
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The benefits of this configurator will be demonstrated by its rapid responsiveness 
and the quality of decisions it facilitates. With this prototype system, the customer 
should be able to gain an accurate understanding of the anticipated system price and 
its complexity before the session of the quotation process has concluded. Further ad- 
justments to the requirements can be explored during the very same conversational 
session. This should reduce a two-week process with multiple tedious back and forth 
requirement-elicitation sessions (as shown in Fig. 2) into a single transparent session 
in which requirements and potential solutions can be communicated effectively and 
dynamically. Furthermore, the session can be concluded with the signing of a hand- 
over confirmation form. Requirements elicitation is central to their business process 
landscape; its successful completion determines and triggers all necessary business 
processes. 

As of recent, we have received a rich and qualitative bounty of new data that we 
aim to utilize for the creation of a new prototype of superior sophistication. Our ap- 
proach is inspired by the designs of the ‘Actionable Cognitive Twin’ (ACT) proposed 
by [9]. ACT establishes a solid foundation that enhances information transparency, de- 
cision support, and agility, while also ensuring that workers actively participate in the 
shaping of business processes, thereby driving innovative initiatives. This integrated 
approach aligns with the goal of effectively utilizing digitalized organizational domain 
knowledge and fostering a culture of data-mindedness and innovation: 
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e It captures both explicit and tacit expert knowledge within Wiewald, providing a 
means for systematic knowledge management and its explicit integration into their 
business process model; leveraging ontologies, knowledge graphs, and embeddings 
to externalize implicit information, thereby fostering a comprehensive understan- 
ding of the business processes. This integration of knowledge promotes effective 
‘knowledge capture’ and ‘-preservation’, ensuring that organizational expertise is 
embedded within the company’s business operations. 

e Furthermore, ACT serves as a holistic measure for innovation management, en- 
couraging active involvement from workers, especially during the structuring and 
modeling of business data and processes, and facilitating worker-driven initiatives 
for innovation. This empowers employees to contribute their valuable insights and 
ideas, creating an environment that fosters collaboration and drives innovation wit- 
hin the organization. Beyond promoting worker empowerment, the implementation 
of ACT enables the adoption of Al-solutions to further enhance innovative organi- 
zational capabilities 


To implement ACT, we would employ a data-driven approach, that leverages natural 
language processing (NLP) techniques such as entity recognition, relationship ex- 
traction, and semantic parsing. Through these techniques, we would extract relevant 
information from collected data and utilize neural network-based ontology learning 
methods to automatically learn ontologies from structured data and textual resources. 
These ontologies capture complex relationships and hierarchical structures, providing 
a foundation for knowledge representation. Furthermore, the ontological learning pro- 
cess would include the creation of an ontological embedding, which represents the 
distilled information from the learned ontology. This embedding serves as valuable 
information for an informed neural network (INN), guiding its understanding and rea- 
soning capabilities. 

The INN, equipped with the ontological embedding, is responsible for the capture 
and analysis of organizational data, leveraging the knowledge captured in the embed- 
ding to interpret and make sense of the incoming data. A knowledge graph serves as 
an instance of the learned ontology and is dynamically populated and updated by the 
INN. By populating the knowledge graph with relevant data from real-time sources 
(like the requirements elicitation form), the INN would create an interconnected re- 
presentation of the acquired information. 

The knowledge graph, shaped by the INN, provides a structured and comprehen- 
sive view of the information captured from the interactive data streams, representing an 
integrated view of the entire organization’s knowledge. Therefore, it should enable sys- 
tematic knowledge management, facilitate decision-making, and foster innovation wit- 
hin the organization. Furthermore, continuous refinement and validation of the domain 
knowledge ontology, knowledge-graphs, and ontological embeddings, in collaboration 
with domain experts, will be needed to ensure their accuracy, relevance, and coverage. 

By implementing the configurator as an instance of ACT, an integrated approach, 
combining ontological learning, ontological embedding, and INN-driven knowledge 
graph population, we aim to unlock new possibilities for effective knowledge repre- 
sentation, reasoning, and information retrieval for the task of compressor system con- 
figuration at Wiewald. 
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Manufacturing and factories 
of the future 


Abstract. The papers of this section deal with developments and potentials for 
manufacturing industry that come with AI. The contributions highlight the production 
of the future and topics that shape the innovation process, such as Industry 4.0, the 
Internet of Things (IoT) and robot-assisted automation. Examples of the use of Al in 
production, process optimization and quality control are presented. 
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Abstract. This paper presents the research project “KI_eeper — Know-how to 
keep” and a first technical research. In this project, tacit experiential knowledge 
of employees is to be recorded and processed by using artificial intelligence to 
make it available to inexperienced workers due to a digital assistance system. 

First, the difference between tacit and explicit knowledge is briefly explai- 
ned. The existing case studies will then be examined in more detail and it will 
be explained why artificial intelligence is necessary for a general solution to 
identify and storage knowledge in this research project. 

Subsequently, various machine learning models and algorithms will be 
considered, which could be used for a potential technical solution. 


Zusammenfassung. Die Babybommer-Generation der 1960er Jahre wird 
in den kommenden Jahren verrentet. Dadurch verlieren Unternehmen lang- 
jährige erfahrene Beschäftigte und auch deren Wissen, wenn nicht effiziente 
Lösungen gefunden werden, dieses zu identifizieren, zu speichern und zu 
transferieren. Dieser Herausforderung widmet sich das Forschungsvorhaben 
„Kl_eeper — Know-how to keep“. Im Projekt wird erforscht, inwiefern künst- 
liche Intelligenz Möglichkeiten eröffnet, um das implizite Erfahrungswissen 
von Beschäftigten automatisiert im Arbeitskontext zu erfassen, zu verarbeiten 
und zu transferieren. Aktuelle Ansätze des Wissenstransfers sind aufwendig 
und häufig auch mit hohen Kosten verbunden. Dabei hat jedoch eben das im- 
plizite Wissen große Relevanz für Unternehmen, welche für kleine und mittlere 
Unternehmen (KMU) noch stärker ausgeprägt ist. 

Am Ende des Projektes soll ein digitales Assistenzsystem entstehen, wel- 
ches das gesammelte und ausgewertete implizite Wissen von Erfahrungsträgern 
allen Beschäftigen zugänglich macht und diese somit bei der Ausführung 
ihrer Tätigkeiten bedarfsgerecht unterstützt. Dabei soll zunächst der Fokus 
auf Tätigkeiten in der Produktionsfertigung gelegt werden. Aufgrund der gro- 
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© Der/die Autor(en) 2024 

C. Zinke-Wehlmann und J. Friedrich (Hrsg.): First Working Conference on Artificial 
Intelligence Development for a Resilient and Sustainable Tomorrow, AIDRST 2023, 
Informatik aktuell, S. 143-151, 2024. https://doi.org/10.1007/978-3-658-43705-3_11 


®) 


Check for 
updates 


144 C. Prange et al. 


Daten, soll eine allgemeingültige Lösung für Verarbeitung der Daten mittels 
der künstlichen Intelligenz gewählt werden. In dieser Veröffentlichung sollen 
ausgewählte Algorithmen betrachtet werden, welche für eine potenzielle tech- 
nische Lösung verwendet werden können. 


Keywords: Knowledge storage - Application and case study - Human-centered 
approaches of Al design & development - Artificial intelligence 


1 Introduction: KI_eeper — Know-how to keep 


The research project “KI_eeper — Know-how to keep”, founded by the German Federal 
Ministry of Education and Research (BMBF), consists of a consortium of research in- 
stitutions, developers and two companies from the metal and electrical industry in Ger- 
many. These provide case studies on which the technical solution aimed in the project 
is to be developed and tested. The aim of this project is to identify, safe and transfer the 
tacit experiential knowledge of long-term employees by using artificial intelligence. 
Knowledge can generally be distinguished as tacit and explicit. Explicit know- 
ledge is understood as knowledge that can be clearly communicated. It can be expres- 
sed in words and numbers, as well as passed on to others as, for example, data or 
processes [1]. Tacit knowledge, on the other hand, is knowledge that cannot be fully 
or adequately communicated in language by the outgoing person. This knowledge is 
often unknowingly embodied in the minds of employees and is manifested in intui- 
tive behavior [2]. Activities do not always consist of explicit or tacit knowledge alone, 
but of the mixture of both. The more tacit knowledge is required, the more difficult 
it is to record and transfer knowledge to another person. Tasks with high proportion 
of tacit knowledge include activities with complex procedures and non-standardized 
processes [3], which can often also be found in manufacturing companies. According 
to North [4], a basic prerequisite for the competitiveness of a company is that emp- 
loyees can apply their experiential knowledge by training-on-the-job. If a long-term 
experienced employee leaves a company without passing the tacit knowledge, this can 
result in a loss of competitiveness. On the other hand, successful handling of existing 
knowledge in a company can even lead to an increase in competitiveness [5]. 
Knowledge management is concerned with how to transfer knowledge to other 
employees within an organization to prevent the disadvantages described by North. 
Approaches such as coaching, mentoring or training offer the opportunity to pass on 
tacit specialist knowledge [6]. Due to the demographic change, such concepts can 
often no longer be pursued, as many specialist positions are not filled and thus the di- 
rect transfer of the necessary knowledge is no longer possible. An increasing number 
of companies are having problems recruiting skilled workers, because the shortage of 
skilled workers is being acute in manufacturing companies [7]. In addition, the trans- 
fer of knowledge is hindered by language barriers of employees with migration back- 
ground. Therefore, a solution should be found how valuable knowledge can be ab- 
sorbed and stored as efficiently as possible. This valuable implicit knowledge should 
subsequently be made available to unskilled workers by means of assistance systems. 
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2 Use Cases 


In the current context of demographic change, the aim of the research project is to re- 
cord tacit knowledge and convert it into explicit knowledge which can then be presen- 
ted to all employees. In particular, new inexperienced employees should be able to be 
trained more quickly in this way. This is to be elaborated based on two very different 
case studies. The use cases in the companies have been selected in such a way that the 
input and output for the intended assistance system differ greatly. In addition, the out- 
puts of the assistance system should be selected in such a way that they are presented 
as simply as possible, so that people with language barriers or semi-skilled employees 
can also use the system and manage the working process professionally. 


2.1 First Use Case: Straightening Flat Steel 


For the first use case, an activity was chosen that requires a high degree of tacit know- 
ledge on the part of an individual person. This involves the manual straightening of 
flat steel, which can be available in different dimensions and material properties. The 
raw material can be up to three meters long, hardened or non-hardened and weigh 
up to 25 kg [8]. For the straightening itself, an oscillating hydraulic joining press is 
used, in which only the ram adjustment is infinitely settable. The change in the ram 
adjustment results in a change in the force acting on the flat steel. According to their 
own statements, the experienced employees at this workstation cannot describe how 
to straighten the flat steel. They work intuitively, which is derived from their long- 
term experience at this station. They cannot convey why they use a certain force to 
straighten at certain points. They feel the rhythm of the press. They intuitively recog- 
nize which of the two edges can be straightened best. They adjust the pressure without 
having to look at the display. They can see with their eyes whether the flat bar still has 
a slight deflection. That is why it takes at least one years and more to learn this job. 

The actual and target condition of the flat steel are used as the first input para- 
meter. The target condition can be retrieved digitally from technical drawings. The 
actual condition of each flat steel is to be recorded with the use of a scanning sensor. 
For the training of the AI, the setting of the path limitation is also required. This can 
be retrieved digitally from the hydraulic press. Furthermore, the exact positioning of 
the flat steel in relation to the pressure point on the press is required so that the tacit 
knowledge can be processed by the AI. At present, a combination of distance sensors 
and the application and reading of a vernier is required for this. Thus, positioning is 
chosen as the third input variable for the flat steel. From these three input variables, 
the artificial intelligence is to calculate the correlation between the acting force and 
the pressure position on the bending of the flat steel. This correlation is to be made 
available to the inexperienced worker in the form of position data — again with the dis- 
tance sensors and the vernier — as well as a setting suggestion for the path limitation. 

Later, during the standard operation of the assistance system, only the actual and 
target states are to be used as input parameters. However, position data and path limi- 
tation settings are still to be defined as output parameters. 
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2.2 Second Use Case: Sub-Process of a Surface Technology System 


The surface technology system of the other applications company was selected as the 
other use case. This is a plant process in which sheet metal parts are coated by means 
of powder coating and wet coating. In this process, the components are first clamped 
in a frame, which is then hung on a carriage of a rail system. This rail system trans- 
ports the components to the individual stations of the system: cleaning, coating, cu- 
ring, and hanging off with quality control. The complete system can be divided into 
several subsystems. The first station at the beginning of the process, where the com- 
ponents are hung up and the machine is controlled/operated, has the most influence on 
the efficiency and workload of the other subsystems. At this point, not only the frame 
is selected to match the components, but also the number of items, the positioning, the 
orientation of the components and the clamping elements used as all as the masking 
positions are determined. The masking serves to ensure that desired areas are not coa- 
ted, e.g., threats or threaded holes. 

The number of different coating products is higher than 2600 with a total of 250 
various colors. Actual there is not much information/instruction on how the compo- 
nents are to be positioned on the frames, which auxiliary materials are to be used or 
which masking measures are to be taken. This information can be partially derived 
from existing technical drawings — including the masking measures. This results in 
the target state of the technical drawings as the first input variable for the training of 
the artificial intelligence. Furthermore, the AI must be fed with the experiential know- 
ledge of the employees. For this purpose, an input is required where the employees 
enter the type of suspension, number of components, etc. In addition, a direct link 
between the AI and the technical drawings should be established at this point. In this 
case, a correlation to the recorded data as well as the target state should also be de- 
fined. As a third input variable, an image of the suspended components is to be re- 
corded after the suspension process. This should be further associated with the other 
input variables. During the recording, not only the individual carriages are to be consi- 
dered, but also their sequence. Since different sized components require various times 
during powder coating as well as other baking times, a certain sequence of diverse 
items must be adhered to maintain the desired cycle time. Currently, this planning of 
the sequence is only possible through the experience of longtime employees. 

After the data training, the cognitive assistance system should provide all emp- 
loyees with suggestions regarding the hanging process when retrieving certain articles 
as an output. Furthermore, the sequence of articles to be processed should be determi- 
ned and communicated by the artificial intelligence. 


3 Methodological Approach 


In order to find suitable AI models for the use-cases described, some criterion must 
first be selected so that a more detailed selection can be made. 

At the time of this publication, the technical solutions only exist as concepts, 
which is why it is not yet possible to make a more precise statement regarding the 
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sensor technology to be selected. However, IoT sensors will be used in both applicati- 
ons. Therefore, the selected model must be able to handle sensor data in general. 

It was also determined that the model must be as transparent as possible or present 
data in such a way that an employee can cognitively grasp the learned properties of 
the model. Therefore, the internal processes of the model should have as little comple- 
xity as possible so that every employee can process the results of the AI. This results 
in complexity as a further criterion for selection. For the same reason, it must also be 
possible to process the interactions between the input variables when capturing the 
tacit knowledge. Considering these two prerequisites, it becomes clear that an explai- 
nable AI is being searched for, since the actual calculation processes running in the 
background are not available as a so-called “black box”, but as a “white box”. These 
processes are therefore available to the user and can be understood. This results in 
another positive aspect: from a socio-technical point of view, prejudices of employees 
could be reduced if it is recognizable what the actual model calculates or works with 
the data. 

The scalability of the AI model also plays a major role. A generalist approach can 
result in both large and small amounts of data for different applications. Thus, the tar- 
geted AI must also be able to cope with large databases. In addition, accuracy is used 
as a criterion for the choice of model to obtain the highest possible resolution of the 
implicit knowledge. 

To obtain a suitable selection of AI models, reference databases such as IEEE 
Xplore and ScienceDirect were accessed. These databases were first roughly sear- 
ched for machine learning approaches which were rated using evaluation criteria. The 
generated data set was compared with the previously described criteria, e.g. that the 
model is explainable. Some of the resulting algorithms will be briefly presented in the 
following chapter. 


4 Presentation of Different AI Models 


Due to the variability of the two use cases, the biggest challenge in this project is 
to develop a universal Al-based solution, to capture the tacit knowledge of the expe- 
rienced employees. Although in both cases there is digital data available for training 
that can be directly defined as input into the AI. The type of input of the experiential 
knowledge is very different. In the first case, the sensor system captures the know- 
ledge completely passively. Whereas in the second use case, although certain data is 
passively recorded by taking photos and by an active input. The employees enter in- 
formation into the technical system at the beginning of the technology introduction, 
e.g., using selection menus and images in one of the use cases. Over time, the sys- 
tem has collected so much data that employees have to actively enter less and less 
information. The system should then be able to make suggestions for improved task 
execution based on the collected data. In the following, some viable models that were 
identified due to research will be briefly presented. 
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4.1 Knowledge-Base Building Procedure 


Huang et al. [10] addressed the involvement of the human expert knowledge in digi- 
tal image processing using a machine learning method to build a knowledge-based 
model. They assume that each object can be represented by its attribute value class 
vector, such as [attribute a_l,..., attribute a_n, class - i]. Therefore, they chose a sub- 
set of data, S, as a representation of data, so that this subset of data includes all pos- 
sible classes. Then, a decision algorithm is applied to the subset of data to partition S 
into subsets of S_1, S_2, ..., S_n based on some production rules. These production 
rules express that the object can be assigned to a certain class if the rules are follo- 
wed. For instance: if attribute 1 > 80 and attribute 2 < 10 then object will be belong to 
class 2. 


4.2 GAMI-Net 


The fundamental idea of the GAMI-Net described by Yang et al. [9] is to capture the 
main effects and pairwise interactions of the model inputs. From this point of view, 
the structure of the GAMI-Net consists of two disjoint networks “Main Effects Mo- 
dule” and “Pairwise Interactions Module”, each performing its own tasks. The Main 
Effects Module tries to capture the nonlinear main effects of the model inputs using 
so-called subnetworks. Each of the subnetworks gets one input and propagates it 
through intermediate layers to calculate one output; while the pairwise interactions 
get two inputs to capture the partwise interactions of the model inputs. Hence, what 
distinguishes these two modules from each other is the number of input neurons in 
their input layers. At last, a linear combination of the modules outputs forms the final 
output. At last, a linear combination of the module outputs forms the final output. The 
following Fig. 1 depicts the structure of the GAMI-Net. 
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Fig. 1 GAMI-Net architecture based on [9] 
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4.3 Bayesian Rule List 


The Bayesian Rule List by Letham et al. [11] is about the generation of a list of rules 
for classifying data. The model consists of two main steps: 


1. Obtaining frequent patterns within the dataset using the FP-tree (frequent-pattern 
tree) algorithm. 
2. Learning of a decision list by selection of the previously obtained patterns. 


Thus, the first step is to learn frequently occurring patterns. The authors use the FP- 
tree algorithm as an example, but according to them it also works with all other algo- 
rithms that extract patterns in data sets. Accordingly in the further process a general 
representation of the finding patterns, which also happens with the FP-tree algorithm. 
Due to the more detailed explanation of FP-tree will be omitted. Patterns are here both 
the characteristics of a property within a data set as well as the combination of charac- 
teristics. The modeler determines when a data pattern occurs frequently. 


4.4 Deep Learning for Case-Based Reasoning through Prototypes 


Li etal. [12] attempted to present a neural network architecture (Deep Learning for 
Case-Based Reasoning through Prototypes - DLCBRP) through which without using a 
second model an explanation for each prediction could be possible. To do this, they pro- 
posed an encoder and a decoder. During training, the data set is processed by the encoder 
in such a way that the dimension of the input is reduced and useful properties for a pre- 
diction are retained. This reduced training data set is then used to learn prototype vectors. 
In this context, a prototype means that it is as close or identical to an observation of the 
training data set. The learned prototype vectors, whose validation might be visualized by 
an encoder network, reside in the prototype layer, which is then connected to the next 
fully connected layer by weights. Finally, by normalizing the weighted sum, a softmax 
layer output is given, which represents a probability distribution of all classes (Fig. 2). 
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Fig. 2 DLCBRP-architecture based on [12] 
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5 Discussion 


In the context of machine learning, not only the choice of an adequate approach 
should be addressed, but also the data that is fed into a model is most important. In 
the described cases, there is tacit knowledge that someone has acquired over years of 
working in a company. Collecting this tacit knowledge is one of the most challenging 
tasks, as any kind of sensitive knowledge must be taken into account in terms of the 
overall data quality. Hence, the model will learn from its training data more effecti- 
vely if more correct training data can be provided. It should be considered that this 
kind of knowledge may not always can be expressed in form of data. 

Another aspect that should be addressed is the explainability of the chosen mo- 
dels. After all, in the context of preparing or teaching someone who has not been 
familiar with a system, there is this expectation that after some hours or days they 
should be able to perform their work independently without the help of supervisors 
or others. Once a suitable model has been trained, it should be able to interpret which 
components or how much of each component is involved in a final product. This paper 
has therefore first tried to select some of the approaches that belong to the explainable 
models. 

The AI models described before have different properties. However, none of the 
discussed algorithms provides a universally valid solution for the inclusion of tacit 
knowledge. With the paper, an intermediate goal has been achieved in which algo- 
rithms for the use case have been investigated. It is evident that further work is requi- 
red to achieve the overall project goal. 
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Abstract. Artificial intelligence (AD is popular for applications in image or 
natural-language processing, but Al can also be used to learn complex relations 
in production processes. For example, an AI can predict product quality based 
on process data during the production. In this paper, we present an application 
of AI to estimate structural properties of additive manufactured components in 
real-time. Occurring imperfections, such as air inclusions in the component, 
are considered and evaluated, since these have a significant influence on the 
quality of the component. This approach combines finite element (FE) simula- 
tion and machine learning: based on FE simulations, a neural network is trai- 
ned to represent the relation between imperfections and the robustness of the 
component. To predict the impact of imperfection in real-time, monitoring sys- 
tems are used to detect anomalies during the printing process, which are indi- 
cations for imperfections in the additive manufactured component. Afterwards, 
the trained model is used to evaluate the impact of the detected anomalies to 
the component quality. This application of AI has a great potential to improve 
the additive manufacturing process itself and simplifying the approval of addi- 
tively manufactured components. 


Zusammenfassung. Die bekanntesten Einsatzgebiete von künstlicher In- 
telligenz (KI) sind zurzeit Bild- und Sprachverarbeitungen. Im Gegensatz 
dazu liegt der Fokus hier auf dem Lernen komplexer Zusammenhänge in 
Produktionsprozessen: untersucht wird ein KI-Anwendungsfall, in dem die 
Qualität des Endproduktes basierend auf Prozessdaten, die während der Pro- 
duktion erhoben werden, vorhergesagt wird. Mit Hilfe Neuronaler Netze wird 
die Belastbarkeit von additiv gefertigten Bauteilen in Echtzeit bewertet. Dazu 
werden Imperfektionen, wie z. B. Lufteinschlüsse im Bauteil, berücksichtigt, 
da diese einen erheblichen Einfluss auf die Qualität des Bauteils haben. Das 
heißt, basierend auf Prozessdaten des additiven Fertigungsprozesses werden 
potenzielle Imperfektionen im Bauteil detektiert und in eine dreidimensionale 
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Repräsentation des Bauteils eingefügt. Mit Hilfe des trainierten Neuronalen 
Netzes werden dann Festigkeitskennwerte für das Bauteil bestimmt, die Auf- 
schluss über die Qualität geben. 

Zum Trainieren eines Neuronalen Netzes ist ein großer Trainingsdatensatz 
notwendig. Diese Daten für diesen Anwendungsfall experimentell zu generieren 
würde bedeuten, dass sehr viele Bauteile mit Imperfektionen gefertigt und an- 
schließend geprüft werden müssten. Um diesen zeitaufwendigen und unwirt- 
schaftlichen Prozess zu umgehen, werden Finite-Elemente-Simulationen zur Er- 
stellung der Datenbasis genutzt. Das heißt, es werden Finite-Elemente-Modelle 
des Bauteils erzeugt, in die Imperfektionen künstlich eingebracht werden. Mit nu- 
merischen Simulationen können anschließend die Festigkeitskennwerte des Bau- 
teils ermittelt werden. Da diese Simulationen zeitaufwändig sind, ist eine Echt- 
zeit-Anwendung nicht möglich. Stattdessen trainieren wir mit den Ergebnissen der 
Simulationen ein Neuronales Netz, dass die Simulationsergebnisse vorhersagen 
soll. Ein solches Neuronales Netz kann dann während des Produktionsprozesses 
genutzt werden, um in Echtzeit die Auswirkungen der detektierten Imperfektionen 
auf die Qualität zu beurteilen. Dadurch können mangelhafte Bauteile aussortiert 
und eine gleichbleibende Qualität gewährleistet werden. Es ist sogar möglich, den 
Druck mangelhafter Bauteile abzubrechen, was Zeit und Ressourcen spart. 


Keywords: Machine Learning - Neural Networks - Finite Element Method - 
Additive Manufacturing - Quality Assurance 


1 Introduction 


Machine learning and artificial intelligence (AI) applications have evolved signifi- 
cantly in the past few years. One field of application is finding relations in large data- 
sets and learning the underlying unknown complex mathematical functions. In this 
paper, we present an Al-based approach that enables quality assurance in an additive 
manufacturing (AM) process. AM, or 3D printing, is a category of production pro- 
cesses where components are created layer wise. It enables the production of highly 
complex components of any geometry and even the design of internal structures such 
as cooling channels or injection nozzles. This design freedom allows to produce light- 
weight and cost-effective components, which makes this technology very attractive 
for the aerospace industry. Since AM is a relatively new technology, there is much 
less experience compared to traditional manufacturing processes, e.g., in terms of the 
impact of disturbances in the production process on the component quality. Conse- 
quently, in order to be allowed to install an AM component, e.g., in an aircraft, an 
approval process must be passed consisting of a lot of destructive and non-destruc- 
tive tests. More precisely, destructive tests are required to verify general material pro- 
perties and the strength of a component. Further, non-destructive testing is required 
to examine each component individually and, if necessary, to initiate post-processing 
steps for quality assurance. For example, a computer tomography (CT) scan is used to 
detect imperfections. Unfortunately, these needed tests and post-processing steps are 
very time-consuming and expensive. To reduce this disadvantage of the AM proces- 
ses, we will show in this paper how production data obtained during the printing pro- 
cess can be used to assure component quality which will decrease the demand of tests 
and make the approval process more efficient. 
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2 Technology Background and State of the Art 


In this paper, we specifically refer to the AM process laser powder bed fusion (LPBF), 
where components are produced layer wise by melting a fine-grained metal powder 
with a high-power laser beam: A three-dimensional component design is divided into 
several layers of a specific thickness in z-direction. For each layer, a uniform powder 
bed is created by the recoating process. The laser beam is directed over this powder 
bed with special scanning mirrors to scan the component areas in the current layer, 
where it melts the powder into a solid metal. The quality of AM components depends 
mainly on the metal powder used and on the printing parameters such as laser power, 
hedge distance, laser speed or layer thickness. However, local defects can occur even 
with well-chosen parameter settings, cf. [1]. The resulting imperfections have been 
shown to cause a negative change in the mechanical structural properties of the final 
component compared to the properties of an optimal component. For example, poro- 
sity contained in the AM component, caused by keyholing or lack of fusion, affects 
dynamic and static properties and can cause anisotropic material behavior, see [2, 3]. 
Furthermore, the surface roughness of AM components particularly affects the fatigue 
properties, cf. [4]. Consequently, comprehensive quality assurance during the produc- 
tion process is essential for the outcome of the AM process. To achieve this, monito- 
ring systems are used to collect information about the manufacturing process from 
which production-related component defects can be derived. The challenge is to de- 
termine the influence of defects on the properties of the manufactured components. 
For this purpose, we present an approach to evaluate the influence of defects to the 
component quality in real-time using Al. 

A crucial point in AM using LPBF is the stable melting of the metal powder. As 
shown, e.g., in [5], the temperature in the melt pool has a significant influence on the 
microstructure of the AM components. For this reason, various monitoring systems 
have been developed to observe the melt pool and several commercial suppliers of 3D 
printers offer monitoring systems for the melt pool or the thermal exposure, such as 
EOS, SLM Solutions, TruPrint, Renishaw and Sigma Additive Solutions. 

We consider in this paper the optical tomography described in [6]. With this moni- 
toring approach, the radiation during the LPBF process is recorded by an off-axial ca- 
mera system, i.e., the camera does not follow the optical path of the fusion laser. The 
light omitted during the melting process consist of three components: reflected laser 
radiation, plasma emission and thermal radiation. A suitable band-pass filter is used to 
record only the thermal radiation. The layer-wise recorded data is visualized color-co- 
ded to show hot-spots and cold-spots. An analysis of AM test specimens has shown a 
correlation between the extreme temperature of spots and material failures. 

Process data generation, i.e., the monitoring of the printing process, has already 
found its way into the industry, whereas the automatic detection of imperfections and 
defects is still the subject of current research. To provide an overview of this ongoing 
research activities, we state in the following a few examples where AI methodologies 
have been applied to identify anomalies in the AM process. 

In [7], an application of supervised machine learning to detect defects is demon- 
strated. The approach is based on images taken for each layer with different lighting 
settings. Features for each location are extracted from these images and classified into 
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anomalous and nominal. In this way, defects in the AM components can be detected 
and localized. A support vector machine is used for the classification. To generate the 
required reference data for supervised learning, anomalies of an AM component are 
detected from CT scans and the associated monitoring images are labelled. 

In [8], three machine learning approaches are applied to detect hot spots based on 
high-speed videos of the printing process. Aim is, to classify between normal and de- 
fect-related brightness evolution. For this purpose, features were extracted from the vi- 
deos, e.g., the mean gradient of the brightness in successive images, the maximum mean 
brightness drop between two consecutive frames, and the shape and size of the bright re- 
gion. Then, these features are used to classify the video snippets by a support vector ma- 
chine or by a neural network. Additionally, k-means is applied to cluster the development 
of the mean brightness. All three methods have shown an accuracy of more than 90 %. 

However, the research results just listed are always only a detection of defects 
or anomalies. An evaluation of their impact on the component’s quality is missing. 
Our vision is to evaluate the component’s quality in real time based on the detected 
defects applying machine learning approaches. For this purpose, we need for many 
components information about their defects and their quality in terms of structural 
properties. To determine the structural properties of a component, it must be proven 
by destroying tests. For example, in tensile testing the component is slowly extended 
until it breaks. With it, e.g., the maximal force that the components can withstand is 
determined. However, this experimental approach requires to produce and destroy a 
large set of components to generate a representative database for training AI methods. 
This is very expensive and time-consuming and practical not feasible. To reduce this 
data generation effort, it was investigated, in a more general application context, whet- 
her numerical simulations can be used to create a representative training database. In 
particular, in [9], it is shown how deep neural networks can be used to estimate biaxial 
stress-strain curves of sheet metals. Based on synthetic generated crystallographic 
texture data, biaxial stress-strain curves were obtained from crystal plasticity-based 
numerical biaxial tensile tests obtained with finite elements (FE) simulations. With 
these numerical test data, deep neural networks were trained. It is shown that their 
predictions are close to the numerical computed test results. However, it is important 
to create a well-grounded data base to applicate the approach to real sheet metals. 

In the following, we will transfer this idea to the AM production process. We pre- 
sent an approach that uses numerical simulations to generate a representative training 
database for AI methods. In detail, we developed an enhanced FE simulation that con- 
siders the material properties of AM components and allows to evaluate the influence 
of imperfections on the component quality. Note that the FE simulations itself could 
not be applied in a real-time quality assurance process because of the large computa- 
tional time needed for one simulation. 


3 Approach 
The monitoring systems currently in use help to manually detect anomalies in the 


melting process which indicate the formation of imperfections in AM components. As 
mentioned above, several commercial 3D printers have integrated monitoring systems 
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to record the melt pool, or the temperatures of the heat affected zone. These systems 
are an important enabler to detect imperfections. To gain greater benefit from monito- 
ring, the detected imperfections must be analyzed in terms of their potential impact on 
component quality. The approach presented here provides such an evaluation. 

Currently, the application of numerical simulation methods, like the FE method, is 
being investigated to determine structural properties of defect-afflicted components. 
In general, numerical simulation methods are based on a discretization of the compo- 
nent to numerically solve the partial differential equations that describe the behavior 
of a component under structural load. However, to obtain reliable results, a fine di- 
scretization is required. For this reason, these numerical methods are computationally 
expensive and cannot be used for a real-time service. 

Another investigated method for monitoring and quality assurance of AM is ma- 
chine learning. One problem of the application of AI methodologies is that usually a 
large number of data sets is required to train an AI. For AM applications, this means 
that many parts must be printed, especially parts with defects, to obtain a representa- 
tive data basis. In other words, to train an AI that learns the relationship between de- 
fect position and structural properties, information about the component condition as 
a result of the manufacturing process (e.g., presence of imperfections) as well as the 
mechanical properties of the components are required. To obtain the mechanical pro- 
perties experimentally, destructive tests must be carried out. Therefore, specimens or 
components have to be manufactured and manually tested one by one. This effort, i.e., 
manufacturing and testing a large number of components to obtain a sufficient data set 


to apply an AI method, is not economically feasible. 
Monitoring data 
of AM process 


Verification 


Use Al 
to detect anomalies 


Feedback 


Apply neural network 
to predict quality 


Fig. 1 Concept for AI based quality assurance in AM 
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In this paper, we combine these two approaches. The use of component-related 
and validated numerical simulation methods offers an alternative to the expensive pro- 
duction and testing step. In our approach, the training data is generated by means of 
FE simulations which compute the mechanical properties of components with imper- 
fections. The data obtained will be used to train a neural network that can predict me- 
chanical properties in real time based on the detected imperfection. 

This approach is illustrated in Fig. | and consists of two steps: training a neural 
network to estimate the quality of AM components (blue colored) and an online appli- 
cation of the trained model during the printing process to predict component strength 
(orange colored). 


3.1 Training of an AI Model 


As it can be seen in Fig. 1, developing a neural network for quality prediction of AM 
components consists of three sub steps: virtual testing, generating training data and 
training itself. Furthermore, experimental tests are used to verify the results of the vir- 
tual tests. 

Numerical simulation methods, i.e. FE simulations, are applied for virtual testing 
of a large set of components with various imperfections scenarios. In detail, the de- 
sign of the component is discretized by defining finite elements, e.g., hexahedral vo- 
lume elements. Based on this discretization, the deformation and stress under load are 
calculated by the numerical solution of partial differential equations. For this purpose, 
the material characteristics must be known, which can be obtained from experimental 
tests. One possibility to compute the effect of pores in the simulation is to define a FE 
model with pores by generating very small finite elements around the pore. However, 
since the pores in AM components are very small, with diameters around 10 um [12], 
a very fine discretization is needed to represent pores which would lead to an extre- 
mely high computational effort. To reduce the overall computational effort, a multi- 
scale method was used [10, 11]. For this purpose, representative volume elements are 
defined for a small piece of material with a pore. The corresponding material charac- 
teristics are computed by preliminary simulations based on very small finite elements. 
These representative volume elements are integrated in the FE model of the construc- 
tion with larger finite elements and represent the imperfections. Subsequently, nume- 
rical simulations are used to predict structural properties and failure behavior of the 
virtually generated AM components with imperfections. However, due to the inhomo- 
geneous material properties the computational effort is still high. With our developed 
FE model generator, the introduction of the imperfections as well as the generation 
of the simulation models can be easily realized. This offers the possibility to generate 
component models with arbitrary and specifically introduced imperfection states and 
to simulate their effects on the structural properties of the component. 

To train a neural network, the FE models with imperfections must be provided 
as training data in combination with the virtual structural test results obtained from 
the simulations. To transform the FE models into input data for a supervised machine 
learning algorithm, the hexahedral volume elements have to be linked to pixels of a 
3D image of the component design. This step is needed, because the elements of the 
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FE model have different sizes and shapes. Furthermore, they are described by their 
corners, which is not intuitive for finding relations between geometrical located de- 
fects and structural properties. In the generated 3D image, the pixels are associated 
with material properties to represent the FE model. For elements affected by defects, 
the material properties of the corresponding representative volume element are used 
for the corresponding pixels. A supervised learning approach is used, where the 3D 
images extracted from the FE models with imperfections are the inputs and the calcu- 
lated structural properties of the FE simulation are the outputs. With these input and 
output data, a convolutional neural network was trained to represent the influence of 
imperfections to the structural properties. 

In further research, experimental testing will be used to validate the virtual tes- 
ting methodology. Note, that this validation requires much less experimental tests than 
generating training data for an AI only with those tests. 


3.2 Online Application of the AI to Predict the Impact of Imperfections 
in Additive Manufactured Components 


As indicated by the orange boxes in Fig. 1, after training a neural network, it can be 
used in the online quality monitoring. This step is also divided into three sub steps: 
Recording monitoring data, detection of anomalies by analyzing monitoring data, and 
estimation of the impact of the detected imperfections by applying the trained neural 
network to predict component’s quality in terms of structural properties. 

During the printing process, monitoring data is recorded, e.g., layer-wise moni- 
toring images, which provide information about the temperatures in the heat affected 
zone, as shown in [6]. The recorded monitoring images have to be analyzed to detect 
anomalies in the printing process which are indications for imperfections like pores. 
Currently, different imaging methods are implemented in the 3D printers, but the mo- 
nitoring data is often not analyzed automatically in real-time. Therefore, an image 
analysis tool needs to be developed to detect anomalies in the provided monitoring 
images. This can be either an Al, e.g., another neural network for image classification, 
or an image processing algorithm. After the automated detection, the anomalies must 
be localized in the printed components. 

Based on the construction, a 3D image of the component is created and the de- 
tected imperfections are localized on the image. Based on this input data, the trained 
neural network predicts the structural properties. If the prediction does not meet the 
requirements, the AM component can be marked as insufficient or even the printing 
process can be aborted, which saves time and energy. 


4 Feasibility Study 


An initial feasibility study with tensile specimens indicated that the developed con- 
cept can be successfully implemented. First, training data for the machine learning 
step was generated using an FE model of a flat tensile specimen, as shown in Fig. 2. 
The hexahedral elements of the FE model are illustrated by the segmentation of the 
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specimen. As it can be seen in Fig. 2, the elements in the examination area, where 
the specimen is thinner and the failure will occur, are significantly smaller than in the 
shoulders, where the specimen is gripped by the testing machine. Each element of the 
FE model is afflicted with material properties, which are amongst others Elastic Mo- 
dulus, breaking point and Poisson’s ratio. The elements with pores have different ma- 
terial properties because a pore reduces the strength. The example of Fig. 2 shows a 
FE model with two sizes of pores coloured in red and green. 


shoulder examination area shoulder 


Fig. 2 FE model of a tensile specimen with imperfections (red and green) 


Fig. 3 3D-image representation of FE model 


In the FE simulation, a virtual tensile test is done whereby the material properties 
of the elements affect the load distribution during the test and with it the strength of 
the whole specimen. One result of the virtual test is the Elastic Modulus for the spe- 
cimen which shows the resistance to being elastically deformed in case of stress. For 
our initial feasibility study, we have used only this result of the FE simulation. 

To train a neural network for the relation between pores in a specimen and Elastic 
Modulus of this specimen, a representative database is required. To generate the trai- 
ning data, several scenarios with different kinds of pores and distributions of pores 
are assumed, i.e., a small set of large pores, several small pores, or pores only in the 
shoulders of the test specimen. For these different scenarios, FE models with ran- 
domly distributed pores are generated and virtually tested. With it, the influence of the 
size and the distribution of the pores on the strength of the specimen is represented in 
the training data. 
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As mentioned above, for the neural network, the FE model is transformed into a 
3D image of the specimen, as shown in Fig. 3, to serve as input data. However, each 
pixel represents the physical characterization of the corresponding element of the FE 
model. In Fig. 3, the dark blue coloured pixels show the material properties of perfect 
printed material, whereas the yellow-coloured pixels represent areas with imperfecti- 
ons and consequently with weaker material properties. In this way, the size difference 
between the finite elements can be taken into account and the localisation of pores as 
well as the closeness to other pores is represented. 

These 3D images and the simulation results, i.e., the Elastic Modulus, which mea- 
sures the stiffness of a material, are the training data. Thereby, the images are the 
input, and the Elastic Modulus is the output, because the neural network should learn, 
how pores in the AM component affect this physical property of the component. Due 
to the input is given as 3D-image, we used 3D convolutional layers for our neural net- 
work. After testing several training architectures and hyperparameters, a suitable neu- 
ral network was obtained. The final architecture of the neural network consists of se- 
veral successive units of one 3D-convolutional layer, one max-pooling layer and one 
batch normalization layer; a layer to flatten the obtained feature tensor; and a dense 
layer with 5 nodes as last hidden layer. With it, about 4000 trainable parameters must 
be determined. Furthermore, our investigations showed that normalization of the input 
data to [0,1] was more suitable than a standardization. For the training, we had about 
350 simulation results. To enlarge the dataset, the specimens are rotated and mirrored, 
which will not affect the simulation results but results in more training data. 
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Fig. 4 Visualization of neural network’s performance 


Figure 4 visualizes the performance of the trained neural network. On the left, the 
prediction and the given output data are compared. Therefore, the data set is sorted by 
the output value, i. e. the Elastic Modulus computed by the FE simulation (black line); 
the orange dots show the corresponding predictions. As it can be seen, the predicti- 
ons are close to the given output values and the general influence of the imperfections 
seems to be represented. On the right of Fig. 4, the distribution of the relative error, 
i.e., the difference between the prediction and the given output value in relation to the 
given output value, is shown. The relative error is generally small. In detail, the me- 
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dian of the relative error is —0.001 % which is close to zero. And for 95 % of the data 
points, the relative error was less than 0.21 %. The maximal relative error was 0.83 %. 


5 Conclusions 


In this paper, we presented an approach to apply Al in quality assurance of an AM pro- 
cess. Thereby, complex FE simulations were used to evaluate the effect of imperfecti- 
ons (i.e., pores) on structural properties of an AM component. To make this important 
information available in real time, the two disciplines FE simulation and machine le- 
arning have been combined. More specifically, a neural network is trained to represent 
the relations between imperfections that occur during production and the structural 
properties of the component. This neural network is used as predictor to estimate on- 
line the impact of the detected anomalies on the robustness of the component. 

In the future, the data base will be expanded to include more complex component 
structures. Here, too, validated simulation tests will be used to determine the influence 
of pore frequencies and positions in the component on its quality. With the increa- 
sing data base, it is expected that not only correlations between pores and quality in 
concrete components can be predicted. With more complex neural network structures, 
more general relationships between, e.g., distances of pores to outer walls, pore sizes 
and wall thicknesses should also be learned. 

We see great potential in this approach for improving the use of AM components. 
The experience gap compared to conventional manufacturing processes is reduced by 
data-driven approaches. Furthermore, since the approach can be validated by real ma- 
terial and component testing, it is possible to reduce the testing effort for approval in 
the future. In this case, significant material and cost savings can be expected. 

The AM process itself can also be improved using this approach. For example, 
parameters can be adjusted or an AM process can be stopped if the AI model predicts 
that the AM component would not pass the approval process. 
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